Orientation Encoding and Viewpoint Invariance in Face Recognition: Inferring Neural Properties from Large-Scale Signals

Viewpoint-invariant face recognition is thought to be subserved by a distributed network of occipitotemporal face-selective areas that, except for the human anterior temporal lobe, have been shown to also contain face-orientation information. This review begins by highlighting the importance of bilateral symmetry for viewpoint-invariant recognition and face-orientation perception. Then, monkey electrophysiological evidence is surveyed describing key tuning properties of face-selective neurons—including neurons bimodally tuned to mirror-symmetric face-views—followed by studies combining functional magnetic resonance imaging (fMRI) and multivariate pattern analyses to probe the representation of face-orientation and identity information in humans. Altogether, neuroimaging studies suggest that face-identity is gradually disentangled from face-orientation information along the ventral visual processing stream. The evidence seems to diverge, however, regarding the prevalent form of tuning of neural populations in human face-selective areas. In this context, caveats possibly leading to erroneous inferences regarding mirror-symmetric coding are exposed, including the need to distinguish angular from Euclidean distances when interpreting multivariate pattern analyses. On this basis, this review argues that evidence from the fusiform face area is best explained by a view-sensitive code reflecting head angular disparity, consistent with a role of this area in face-orientation perception. Finally, the importance is stressed of explicit models relating neural properties to large-scale signals.

[1]  H. Wallach,et al.  Figural aftereffects; an investigation of visual processes. , 1944 .

[2]  R. Desimone,et al.  Stimulus-selective properties of inferior temporal neurons in the macaque , 1984, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[3]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[4]  Edmund T. Rolls,et al.  The neuronal encoding of information in the brain , 2011, Progress in Neurobiology.

[5]  R. Tootell,et al.  An anterior temporal face patch in human cortex, predicted by macaque maps , 2009, Proceedings of the National Academy of Sciences.

[6]  G. Rhodes,et al.  Broadly tuned, view-specific coding of face shape: Opposing figural aftereffects can be induced in different views , 2007, Vision Research.

[7]  Juha Silvanto,et al.  The Causal Role of the Occipital Face Area (OFA) and Lateral Occipital (LO) Cortex in Symmetry Perception , 2015, The Journal of Neuroscience.

[8]  J. Sergent,et al.  Functional neuroanatomy of face and object processing. A positron emission tomography study. , 1992, Brain : a journal of neurology.

[9]  Johan D. Carlin,et al.  Decoding Face Exemplars from fMRI Responses: What Works, What Doesn't? , 2015, The Journal of Neuroscience.

[10]  F. Fang,et al.  Perceptual consequences of face viewpoint adaptation: face viewpoint aftereffect, changes of differential sensitivity to face view, and their relationship. , 2010, Journal of vision.

[11]  Ken Nakayama,et al.  The consequences of subtracting the mean pattern in fMRI multivariate correlation analyses , 2013, Front. Neurosci..

[12]  William Hyde Wollaston,et al.  XIII. On the apparent direction of eyes in a portrait , 1824, Philosophical Transactions of the Royal Society of London.

[13]  Ha Hong,et al.  Explicit information for category-orthogonal object properties increases along the ventral stream , 2016, Nature Neuroscience.

[14]  Luca Passamonti,et al.  Connectivity Analysis Reveals a Cortical Network for Eye Gaze Perception , 2009, Cerebral cortex.

[15]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[16]  Doris Y. Tsao,et al.  Functional Compartmentalization and Viewpoint Generalization Within the Macaque Face-Processing System , 2010, Science.

[17]  Pierre Jolicoeur,et al.  Identification of Disoriented Objects: A Dual‐systems Theory , 1990 .

[18]  Ivo D. Popivanov,et al.  Probabilistic and Single-Subject Retinotopic Maps Reveal the Topographic Organization of Face Patches in the Macaque Cortex , 2014, The Journal of Neuroscience.

[19]  D. Kersten,et al.  Three-dimensional symmetric shapes are discriminated more efficiently than asymmetric ones. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[20]  Daniel D. Dilks,et al.  Differential selectivity for dynamic versus static information in face-selective cortical regions , 2011, NeuroImage.

[21]  Doris Y. Tsao,et al.  A Cortical Region Consisting Entirely of Face-Selective Cells , 2006, Science.

[22]  Doris Y. Tsao,et al.  Intelligent Information Loss: The Coding of Facial Identity, Head Pose, and Non-Face Information in the Macaque Face Patch System , 2015, The Journal of Neuroscience.

[23]  K. Grill-Spector,et al.  The functional architecture of the ventral temporal cortex and its role in categorization , 2014, Nature Reviews Neuroscience.

[24]  G. Rhodes,et al.  View-Specific Coding of Face Shape , 2006 .

[25]  J. Macke,et al.  Neural population coding: combining insights from microscopic and mass signals , 2015, Trends in Cognitive Sciences.

[26]  T. Poggio,et al.  The importance of symmetry and virtual views in three-dimensional object recognition , 1994, Current Biology.

[27]  P. Schyns,et al.  Information and viewpoint dependence in face recognition , 1997, Cognition.

[28]  Alice J. O'Toole,et al.  Dissociable Neural Patterns of Facial Identity across Changes in Viewpoint , 2010, Journal of Cognitive Neuroscience.

[29]  N. Troje,et al.  Illumination-Induced Apparent Shift in Orientation of Human Heads , 1998, Perception.

[30]  H H Bülthoff,et al.  How are three-dimensional objects represented in the brain? , 1994, Cerebral cortex.

[31]  H. Bülthoff,et al.  Face recognition under varying poses: The role of texture and shape , 1996, Vision Research.

[32]  Keiji Tanaka,et al.  Functional architecture in monkey inferotemporal cortex revealed by in vivo optical imaging , 1998, Neuroscience Research.

[33]  A. J. Mistlin,et al.  Visual cells in the temporal cortex sensitive to face view and gaze direction , 1985, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[34]  D. V. van Essen,et al.  Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex. , 1993, Science.

[35]  S. Anstis,et al.  The perception of where a face or television "portrait" is looking. , 1969, The American journal of psychology.

[36]  Heinrich H Bülthoff,et al.  Image-based object recognition in man, monkey and machine , 1998, Cognition.

[37]  Doris Y. Tsao,et al.  Single-Unit Recordings in the Macaque Face Patch System Reveal Limitations of fMRI MVPA , 2015, The Journal of Neuroscience.

[38]  H. P. Op de Beeck,et al.  Representations of Facial Identity Information in the Ventral Visual Stream Investigated with Multivoxel Pattern Analyses , 2013, The Journal of Neuroscience.

[39]  J. Gibson,et al.  Perception of another person's looking behavior. , 1963, The American journal of psychology.

[40]  David L. Sheinberg,et al.  Visual object recognition. , 1996, Annual review of neuroscience.

[41]  N. Logothetis,et al.  Shape representation in the inferior temporal cortex of monkeys , 1995, Current Biology.

[42]  Christopher W Tyler,et al.  Face configuration processing in the human brain: the role of symmetry. , 2007, Cerebral cortex.

[43]  A. Ishai,et al.  Recollection- and Familiarity-Based Decisions Reflect Memory Strength , 2008, Frontiers in systems neuroscience.

[44]  K. Grill-Spector,et al.  Neural representations of faces and limbs neighbor in human high-level visual cortex: evidence for a new organization principle , 2011, Psychological Research.

[45]  A. Caramazza,et al.  Decoding representations of face identity that are tolerant to rotation. , 2014, Cerebral cortex.

[46]  Keiji Tanaka,et al.  Optical Imaging of Functional Organization in the Monkey Inferotemporal Cortex , 1996, Science.

[47]  Thomas Naselaris,et al.  Resolving Ambiguities of MVPA Using Explicit Models of Representation , 2015, Trends in Cognitive Sciences.

[48]  Hisao Nishijo,et al.  Differential characteristics of face neuron responses within the anterior superior temporal sulcus of macaques. , 2005, Journal of neurophysiology.

[49]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[50]  M. Harries,et al.  Viewer-centred and object-centred coding of heads in the macaque temporal cortex , 2004, Experimental Brain Research.

[51]  C. Connor,et al.  Neural representations for object perception: structure, category, and adaptive coding. , 2011, Annual review of neuroscience.

[52]  F. Fang,et al.  The role of gaze direction in face viewpoint aftereffect , 2009, Vision Research.

[53]  K. Grill-Spector,et al.  Electrical Stimulation of Human Fusiform Face-Selective Regions Distorts Face Perception , 2012, The Journal of Neuroscience.

[54]  G. Yovel,et al.  Successful Decoding of Famous Faces in the Fusiform Face Area , 2015, PloS one.

[55]  G. Yovel,et al.  Hierarchical Processing of Face Viewpoint in Human Visual Cortex , 2012, The Journal of Neuroscience.

[56]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[57]  H. Wilson,et al.  Perception of head orientation , 2000, Vision Research.

[58]  N. George,et al.  Facing the gaze of others , 2008, Neurophysiologie Clinique/Clinical Neurophysiology.

[59]  J. Haxby,et al.  Distributed Neural Systems for Face Perception , 2011 .

[60]  Radoslaw Martin Cichy,et al.  The Neural Code for Face Orientation in the Human Fusiform Face Area , 2014, The Journal of Neuroscience.

[61]  Fernando M Ramírez,et al.  Representational confusion: the plausible consequence of demeaning your data , 2017, bioRxiv.

[62]  Ravi S. Menon,et al.  An fMRI study of the selective activation of human extrastriate form vision areas by radial and concentric gratings , 2000, Current Biology.

[63]  Hisao Nishijo,et al.  Neuronal correlates of face identification in the monkey anterior temporal cortical areas. , 2004, Journal of neurophysiology.

[64]  S. Ullman Aligning pictorial descriptions: An approach to object recognition , 1989, Cognition.

[65]  Frank Tong,et al.  Prevalence of Selectivity for Mirror-Symmetric Views of Faces in the Ventral and Dorsal Visual Pathways , 2012, The Journal of Neuroscience.

[66]  Ingrid R. Olson,et al.  Beyond the FFA: The role of the ventral anterior temporal lobes in face processing , 2014, Neuropsychologia.

[67]  C. Gross Single neuron studies of inferior temporal cortex , 2008, Neuropsychologia.

[68]  Michael J. Tarr,et al.  Task-Specific Codes for Face Recognition: How they Shape the Neural Representation of Features for Detection and Individuation , 2008, PloS one.

[69]  Johan D. Carlin,et al.  The neural basis of eye gaze processing , 2013, Current Opinion in Neurobiology.

[70]  Marlene Behrmann,et al.  Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis , 2011, Proceedings of the National Academy of Sciences.

[71]  Nathan Intrator,et al.  (coarse Coding of Shape Fragments) (retinotopy) Representation of Structure , 2000 .

[72]  M. Ida Gobbini,et al.  Disentangling the representation of identity from head view along the human face processing pathway , 2016 .

[73]  Galit Yovel,et al.  A Revised Neural Framework for Face Processing. , 2015, Annual review of vision science.

[74]  R. Goebel,et al.  Individual faces elicit distinct response patterns in human anterior temporal cortex , 2007, Proceedings of the National Academy of Sciences.

[75]  T. Poggio,et al.  A network that learns to recognize three-dimensional objects , 1990, Nature.

[76]  Kalanit Grill-Spector,et al.  Sparsely-distributed organization of face and limb activations in human ventral temporal cortex , 2010, NeuroImage.

[77]  Christoph Kayser,et al.  Who is That? Brain Networks and Mechanisms for Identifying Individuals , 2015, Trends in Cognitive Sciences.

[78]  Jon Driver,et al.  Parallel computation of symmetry but not repetition within single visual shapes , 1994 .

[79]  C. Darwin The Expression of the Emotions in Man and Animals , .

[80]  Johan Wagemans,et al.  The role of vertical mirror symmetry in visual shape detection. , 2009, Journal of vision.

[81]  N. Kanwisher,et al.  Only some spatial patterns of fMRI response are read out in task performance , 2007, Nature Neuroscience.

[82]  N. Logothetis What we can do and what we cannot do with fMRI , 2008, Nature.

[83]  I. Biederman,et al.  Evidence for Complete Translational and Reflectional Invariance in Visual Object Priming , 1991, Perception.

[84]  Wim Vanduffel,et al.  Symmetry activates extrastriate visual cortex in human and nonhuman primates. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[85]  C Blakemore,et al.  On the existence of neurones in the human visual system selectively sensitive to the orientation and size of retinal images , 1969, The Journal of physiology.

[86]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[87]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[88]  Roberto Caldara,et al.  The Fusiform Face Area responds automatically to statistical regularities optimal for face categorization , 2009, Human brain mapping.

[89]  John A. Pyles,et al.  Dynamic Encoding of Face Information in the Human Fusiform Gyrus , 2014, Nature Communications.

[90]  Essa Yacoub,et al.  Modeling and analysis of mechanisms underlying fMRI-based decoding of information conveyed in cortical columns , 2011, NeuroImage.

[91]  T. Allison,et al.  Face-Specific Processing in the Human Fusiform Gyrus , 1997, Journal of Cognitive Neuroscience.

[92]  M. Hasselmo,et al.  Object-centered encoding by face-selective neurons in the cortex in the superior temporal sulcus of the monkey , 2004, Experimental Brain Research.

[93]  P. Ekman,et al.  Emotion in the Human Face: Guidelines for Research and an Integration of Findings , 1972 .

[94]  F. Fang,et al.  Transfer of the face viewpoint aftereffect from adaptation to different and inverted faces. , 2007, Journal of vision.

[95]  Jeremy Freeman,et al.  Orientation Decoding Depends on Maps, Not Columns , 2011, The Journal of Neuroscience.

[96]  Stephan Eliez,et al.  Effect of head orientation on gaze processing in fusiform gyrus and superior temporal sulcus , 2003, NeuroImage.

[97]  Brittany S. Cassidy,et al.  Lower-Level Stimulus Features Strongly Influence Responses in the Fusiform Face Area , 2010, Cerebral cortex.

[98]  D. Perrett,et al.  Visual neurones responsive to faces in the monkey temporal cortex , 2004, Experimental Brain Research.

[99]  A. Young,et al.  Understanding person perception. , 2011, British journal of psychology.

[100]  A. Young,et al.  Understanding face recognition. , 1986, British journal of psychology.

[101]  Brian A. Wandell,et al.  Predominantly extra-retinotopic cortical response to pattern symmetry , 2005, NeuroImage.

[102]  Michael S. Pratte,et al.  Decoding patterns of human brain activity. , 2012, Annual review of psychology.

[103]  K. Grill-Spector,et al.  fMR-adaptation: a tool for studying the functional properties of human cortical neurons. , 2001, Acta psychologica.

[104]  H. Wilson,et al.  The face viewpoint aftereffect: Adapting to full faces, head outlines, and features , 2012, Vision Research.

[105]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[106]  Johan Wagemans,et al.  Distributed subordinate specificity for bodies, faces, and buildings in human ventral visual cortex , 2010, NeuroImage.

[107]  Andrew W Young,et al.  The importance of internal facial features in learning new faces , 2015, Quarterly journal of experimental psychology.

[108]  Randolph Blake,et al.  The Occipital Face Area Is Causally Involved in Facial Viewpoint Perception , 2015, The Journal of Neuroscience.

[109]  E. Rolls The representation of information about faces in the temporal and frontal lobes , 2007, Neuropsychologia.

[110]  Sheng He,et al.  Viewer-Centered Object Representation in the Human Visual System Revealed by Viewpoint Aftereffects , 2005, Neuron.

[111]  J. Hummel,et al.  The role of attention in priming for left-right reflections of object images: evidence for a dual representation of object shape. , 1998, Journal of experimental psychology. Human perception and performance.

[112]  S. Pollmann,et al.  Retinotopic Activation in Response to Subjective Contours in Primary Visual Cortex , 2008, Frontiers in human neuroscience.

[113]  Brian A. Wandell,et al.  Population receptive field estimates in human visual cortex , 2008, NeuroImage.

[114]  W. Freiwald,et al.  Face Processing Systems: From Neurons to Real-World Social Perception. , 2016, Annual review of neuroscience.

[115]  I. Biederman Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[116]  Frank Tong,et al.  Representational Dynamics of Facial Viewpoint Encoding , 2017, Journal of Cognitive Neuroscience.

[117]  Joel Z. Leibo,et al.  View-Tolerant Face Recognition and Hebbian Learning Imply Mirror-Symmetric Neural Tuning to Head Orientation , 2016, Current Biology.

[118]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[119]  Shimon Ullman,et al.  Face Recognition: The Problem of Compensating for Changes in Illumination Direction , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[120]  David D. Cox,et al.  Opinion TRENDS in Cognitive Sciences Vol.11 No.8 Untangling invariant object recognition , 2022 .

[121]  John-Dylan Haynes,et al.  Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA , 2014, NeuroImage.

[122]  Doris Y. Tsao,et al.  Comparing face patch systems in macaques and humans , 2008, Proceedings of the National Academy of Sciences.

[123]  Rob Jenkins,et al.  Arguments Against a Configural Processing Account of Familiar Face Recognition , 2015, Perspectives on psychological science : a journal of the Association for Psychological Science.

[124]  M. Cline The perception of where a person is looking. , 1967, The American journal of psychology.

[125]  Bruno Rossion,et al.  Does physical interstimulus variance account for early electrophysiological face sensitive responses in the human brain? Ten lessons on the N170 , 2008, NeuroImage.

[126]  G. Yovel,et al.  TMS Evidence for the Involvement of the Right Occipital Face Area in Early Face Processing , 2007, Current Biology.

[127]  Gillian Rhodes,et al.  Higher-level mechanisms detect facial symmetry , 2005, Proceedings of the Royal Society B: Biological Sciences.

[128]  D I Perrett,et al.  Organization and functions of cells responsive to faces in the temporal cortex. , 1992, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.