A bstract The Swampland Distance Conjecture suggests that an infinite tower of modes becomes exponentially light when approaching a point that is at infinite proper distance in field space. In this paper we investigate this conjecture in the Kähler moduli spaces of Calabi-Yau threefold compactifications and further elucidate the proposal that the infinite tower of states is generated by the discrete symmetries associated to infinite distance points. In the large volume regime the infinite tower of states is generated by the action of the local monodromy matrices and encoded by an orbit of D-brane charges. We express these monodromy matrices in terms of the triple intersection numbers to classify the infinite distance points and construct the associated infinite charge orbits that become massless. We then turn to a detailed study of charge orbits in elliptically fibered Calabi-Yau threefolds. We argue that for these geometries the modular symmetry in the moduli space can be used to transfer the large volume orbits to the small elliptic fiber regime. The resulting orbits can be used in compactifications of M-theory that are dual to F-theory compactifications including an additional circle. In particular, we show that there are always charge orbits satisfying the distance conjecture that correspond to Kaluza-Klein towers along that circle. Integrating out the KK towers yields an infinite distance in the moduli space thereby supporting the idea of emergence in that context.