Colloids are valuable model systems to understand the structure and dynamics of matter, explore new self-assembly concepts, and realize advanced materials. Herein, we demonstrate social self-sorting of co-assembled families of colloids by orthogonal host/guest recognition using cyclodextrins. We show that mixtures of four partners self-sort into their respective families without mutual interference. Additionally, the self-assemblies and their interactions are switchable using orthogonal triggers. This study goes beyond previous features of molecular self-sorting, and opens the design space for future self-sorting colloidal systems via rationally designed molecular recognition.