Deep convolutional neural networks in the face of caricature

Real-world face recognition requires us to perceive the uniqueness of a face across variable images. Deep convolutional neural networks (DCNNs) accomplish this feat by generating robust face representations that can be analysed in a multidimensional ‘face space’. We examined the organization of viewpoint, illumination, gender and identity in this space. We found that DCNNs create a highly organized face similarity structure in which identities and images coexist. Natural image variation is organized hierarchically, with face identity nested under gender, and illumination and viewpoint nested under identity. To examine identity, we caricatured faces and found that identification accuracy increased with the strength of identity information in a face, and caricature representations ‘resembled’ their veridical counterparts—mimicking human perception. DCNNs therefore offer a theoretical framework for reconciling decades of behavioural and neural results that emphasized either the image or the face in representations, without understanding how a neural code could seamlessly accommodate both.Human face recognition is robust to changes in viewpoint, illumination, facial expression and appearance. The authors investigated face recognition in deep convolutional neural networks by manipulating the strength of identity information in a face by caricaturing. They found that networks create a highly organized face similarity structure in which identities and images coexist.