Poly(vinyl pyridine) has widely been used as a pH‐responsive polymer to trigger changes in self‐assembly of block copolymer micelles. However, the polymer is known to display toxic features, which limits its ultimate applicability for biological applications. Here, poly(4‐vinyl imidazole) (P4VIm), a much more biocompatible polymer, is used as a pH‐responsive block to modulate the self‐assembly of ABC triblock terpolymers. In this article, the synthesis of the poly(1‐acryloyl fructopyranose)‐block‐ poly(n‐butyl acrylate)‐block‐ poly(4‐vinyl imidazole) (PFruA52‐b‐PBuA300‐b‐P4VIm250) triblock terpolymers is first discussed by sequential reversible addition fragmentation chain transfer (RAFT) polymerization. Subsequently, the structure formation of the triblock terpolymer is elucidated by step‐wise solvent exchange. The polymer readily dissolves in methanol, but self‐assembles into micelles with PBuA cores and mixed shell in methanol–water mixtures. Solvent exchange against buffer solutions of pH 6–6.5 leads to collapse of P4VIm due to deprotonation and induces self‐assembly into caterpillar‐like non‐spherical nanoparticles, most likely via the formation of intermediate Janus particles. The rearrangement into larger hierarchical structure, as seen by small angle X‐ray scattering (SAXS), is found to process within several hours. The article is concluded by demonstrating lower cytotoxicity values for the present polymer in comparison to a structurally analogous triblock terpolymer based on poly(vinyl pyridine).