The Neural Dynamics of Facial Identity Processing: Insights from EEG-Based Pattern Analysis and Image Reconstruction

Abstract Uncovering the neural dynamics of facial identity processing along with its representational basis outlines a major endeavor in the study of visual processing. To this end, here, we record human electroencephalography (EEG) data associated with viewing face stimuli; then, we exploit spatiotemporal EEG information to determine the neural correlates of facial identity representations and to reconstruct the appearance of the corresponding stimuli. Our findings indicate that multiple temporal intervals support: facial identity classification, face space estimation, visual feature extraction and image reconstruction. In particular, we note that both classification and reconstruction accuracy peak in the proximity of the N170 component. Further, aggregate data from a larger interval (50–650 ms after stimulus onset) support robust reconstruction results, consistent with the availability of distinct visual information over time. Thus, theoretically, our findings shed light on the time course of face processing while, methodologically, they demonstrate the feasibility of EEG-based image reconstruction.

[1]  H. Wilson,et al.  fMRI evidence for the neural representation of faces , 2005, Nature Neuroscience.

[2]  T. Valentine The Quarterly Journal of Experimental Psychology Section A: Human Experimental Psychology a Unified Account of the Effects of Distinctiveness, Inversion, and Race in Face Recognition , 2022 .

[3]  P. Schyns,et al.  Measuring Internal Representations from Behavioral and Brain Data , 2012, Current Biology.

[4]  Margot J. Taylor,et al.  Inversion and Contrast Polarity Reversal Affect both Encoding and Recognition Processes of Unfamiliar Faces: A Repetition Study Using ERPs , 2002, NeuroImage.

[5]  Marlene Behrmann,et al.  Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysis , 2011, Proceedings of the National Academy of Sciences.

[6]  G. Rhodes,et al.  Sex-specific norms code face identity. , 2011, Journal of vision.

[7]  Marlene Behrmann,et al.  Feature-based face representations and image reconstruction from behavioral and neural data , 2015, Proceedings of the National Academy of Sciences.

[8]  A. Young,et al.  Contributions of feature shapes and surface cues to the recognition and neural representation of facial identity , 2016, Cortex.

[9]  B. Rossion,et al.  ERP evidence for the speed of face categorization in the human brain: Disentangling the contribution of low-level visual cues from face perception , 2011, Vision Research.

[10]  Thomas Serre,et al.  The Neural Dynamics of Face Detection in the Wild Revealed by MVPA , 2014, The Journal of Neuroscience.

[11]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[12]  Joel Z. Leibo,et al.  The dynamics of invariant object recognition in the human visual system. , 2014, Journal of neurophysiology.

[13]  Alexander Borst,et al.  How does Nature Program Neuron Types? , 2008, Front. Neurosci..

[14]  Doris Y. Tsao,et al.  The Code for Facial Identity in the Primate Brain , 2017, Cell.

[15]  A. Caramazza,et al.  Decoding representations of face identity that are tolerant to rotation. , 2014, Cerebral cortex.

[16]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[17]  J. Tanaka,et al.  A Neural Basis for Expert Object Recognition , 2001, Psychological science.

[18]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[19]  L. Deouell,et al.  STRUCTURAL ENCODING AND IDENTIFICATION IN FACE PROCESSING: ERP EVIDENCE FOR SEPARATE MECHANISMS , 2000, Cognitive neuropsychology.

[20]  A. O'Toole,et al.  Probing the Visual Representation of Faces With Adaptation , 2006, Psychological science.

[21]  D. Plaut,et al.  Spatiotemporal dynamics of similarity-based neural representations of facial identity , 2016, Proceedings of the National Academy of Sciences.

[22]  H. Wilson,et al.  The neural representation of face space dimensions , 2013, Neuropsychologia.

[23]  A. Treves,et al.  Morphing Marilyn into Maggie dissociates physical and identity face representations in the brain , 2005, Nature Neuroscience.

[24]  Garrison W. Cottrell,et al.  What Evidence Supports Special Processing for Faces? A Cautionary Tale for fMRI Interpretation , 2013, Journal of Cognitive Neuroscience.

[25]  Isabel Gauthier,et al.  Are Face and Object Recognition Independent? A Neurocomputational Modeling Exploration , 2016, Journal of Cognitive Neuroscience.

[26]  Terrence J. Sejnowski,et al.  Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis , 2007, NeuroImage.

[27]  Sébastien M. Crouzet,et al.  Fast saccades toward faces: face detection in just 100 ms. , 2010, Journal of vision.

[28]  C. Thomaz,et al.  A new ranking method for principal components analysis and its application to face image analysis , 2010, Image Vis. Comput..

[29]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[30]  Tandra Ghose,et al.  Generalization between canonical and non-canonical views in object recognition. , 2013, Journal of vision.

[31]  James W. Tanaka,et al.  Activation of Preexisting and Acquired Face Representations: The N250 Event-related Potential as an Index of Face Familiarity , 2006, Journal of Cognitive Neuroscience.

[32]  P. Schyns,et al.  Superstitious Perceptions Reveal Properties of Internal Representations , 2003, Psychological science.

[33]  Doris Y. Tsao,et al.  A face feature space in the macaque temporal lobe , 2009, Nature Neuroscience.

[34]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[35]  G. Kreiman,et al.  Timing, Timing, Timing: Fast Decoding of Object Information from Intracranial Field Potentials in Human Visual Cortex , 2009, Neuron.

[36]  A. Burton,et al.  Matching faces for semantic information and names: an event-related brain potentials study. , 2003, Brain research. Cognitive brain research.

[37]  Sidney R. Lehky Fine Discrimination of Faces can be Performed Rapidly , 2000, Journal of Cognitive Neuroscience.

[38]  C. Jacques,et al.  Early adaptation to repeated unfamiliar faces across viewpoint changes in the right hemisphere: Evidence from the N170 ERP component , 2009, Neuropsychologia.

[39]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[40]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[41]  J. Shedden,et al.  Progressive N170 habituation to unattended repeated faces , 2006, Vision Research.

[42]  M. Landy,et al.  The effect of viewpoint on perceived visual roughness. , 2007, Journal of vision.

[43]  A. Burton,et al.  Human brain potential correlates of repetition priming in face and name recognition , 2002, Neuropsychologia.

[44]  G. Yovel,et al.  Diagnosing prosopagnosia: Effects of ageing, sex, and participant–stimulus ethnic match on the Cambridge Face Memory Test and Cambridge Face Perception Test , 2009, Cognitive neuropsychology.

[45]  David F. Marks,et al.  New directions for mental imagery research. , 1995 .

[46]  A. Norcia,et al.  A Representational Similarity Analysis of the Dynamics of Object Processing Using Single-Trial EEG Classification , 2015, PloS one.

[47]  D. Plaut,et al.  Face-Space Architectures , 2013, Psychological science.

[48]  Donald A. Jackson PROTEST: A PROcrustean Randomization TEST of community environment concordance , 1995 .

[49]  David A. Tovar,et al.  Representational dynamics of object vision: the first 1000 ms. , 2013, Journal of vision.

[50]  A Mouraux,et al.  Across-trial averaging of event-related EEG responses and beyond. , 2008, Magnetic resonance imaging.

[51]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[52]  Marcel van Gerven,et al.  MEG-based decoding of the spatiotemporal dynamics of visual category perception , 2013, NeuroImage.

[53]  John A. Pyles,et al.  Dynamic Encoding of Face Information in the Human Fusiform Gyrus , 2014, Nature Communications.

[54]  Stefan Haufe,et al.  Single-trial analysis and classification of ERP components — A tutorial , 2011, NeuroImage.

[55]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[56]  I. Blank,et al.  The structure of face--space is tolerant to lighting and viewpoint transformations. , 2011, Journal of vision.

[57]  Radoslaw Martin Cichy,et al.  Resolving human object recognition in space and time , 2014, Nature Neuroscience.

[58]  Richard F Murray,et al.  Classification images: A review. , 2011, Journal of vision.

[59]  M. Giese,et al.  Norm-based face encoding by single neurons in the monkey inferotemporal cortex , 2006, Nature.

[60]  C. Jacques,et al.  The time course of the inversion effect during individual face discrimination. , 2007, Journal of vision.

[61]  Brice A. Kuhl,et al.  Neural portraits of perception: Reconstructing face images from evoked brain activity , 2014, NeuroImage.

[62]  Susan G. Wardle,et al.  Decoding Dynamic Brain Patterns from Evoked Responses: A Tutorial on Multivariate Pattern Analysis Applied to Time Series Neuroimaging Data , 2016, Journal of Cognitive Neuroscience.

[63]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[64]  Michael C. Mangini,et al.  Do Humans and Baboons Use the Same Information When Categorizing Human and Baboon Faces? , 2006, Psychological science.

[65]  Matthias Niemeier,et al.  The time course of individual face recognition: A pattern analysis of ERP signals , 2016, NeuroImage.

[66]  H. P. Op de Beeck,et al.  Representations of Facial Identity Information in the Ventral Visual Stream Investigated with Multivoxel Pattern Analyses , 2013, The Journal of Neuroscience.

[67]  Tal Makovski,et al.  The visual attractor illusion. , 2010, Journal of vision.

[68]  Yaroslav O. Halchenko,et al.  Brain Reading Using Full Brain Support Vector Machines for Object Recognition: There Is No Face Identification Area , 2008, Neural Computation.

[69]  K. Nakayama,et al.  The Cambridge Face Memory Test: Results for neurologically intact individuals and an investigation of its validity using inverted face stimuli and prosopagnosic participants , 2006, Neuropsychologia.

[70]  T. Valentine,et al.  Recognizing Unfamiliar Faces: The Effects of Distinctiveness and View , 1999, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[71]  David M. Groppe,et al.  Exemplar selectivity reflects perceptual similarities in the human fusiform cortex. , 2014, Cerebral cortex.

[72]  E. Erdfelder,et al.  Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses , 2009, Behavior research methods.

[73]  C. Mondloch,et al.  The timing of individual face recognition in the brain , 2012, Neuropsychologia.

[74]  P. Bennett,et al.  Inversion Leads to Quantitative, Not Qualitative, Changes in Face Processing , 2004, Current Biology.