Face perception is crucial to social interactions, yet people vary in how easily they can recognize their friends, verify an identification document or notice someone’s smile. There are widespread differences in people’s ability to recognize faces, and research has particularly focused on exceptionally good or poor recognition performance. In this Review, we synthesize the literature on individual differences in face processing across various tasks including identification and estimates of emotional state and social attributes. The individual differences approach has considerable untapped potential for theoretical progress in understanding the perceptual and cognitive organization of face processing. This approach also has practical consequences — for example, in determining who is best suited to check passports. We also discuss the underlying structural and anatomical predictors of face perception ability. Furthermore, we highlight problems of measurement that pose challenges for the effective study of individual differences. Finally, we note that research in individual differences rarely addresses perception of familiar faces. Despite people’s everyday experience of being ‘good’ or ‘bad’ with faces, a theory of how people recognize their friends remains elusive.