Brain lateralization of holistic versus analytic processing of emotional facial expressions

This study investigated the neurocognitive mechanisms underlying the role of the eye and the mouth regions in the recognition of facial happiness, anger, and surprise. To this end, face stimuli were shown in three formats (whole face, upper half visible, and lower half visible) and behavioral categorization, computational modeling, and ERP (event-related potentials) measures were combined. N170 (150-180 ms post-stimulus; right hemisphere) and EPN (early posterior negativity; 200-300 ms; mainly, right hemisphere) were modulated by expression of whole faces, but not by separate halves. This suggests that expression encoding (N170) and emotional assessment (EPN) require holistic processing, mainly in the right hemisphere. In contrast, the mouth region of happy faces enhanced left temporo-occipital activity (150-180 ms), and also the LPC (late positive complex; centro-parietal) activity (350-450 ms) earlier than the angry eyes (450-600 ms) or other face regions. Relatedly, computational modeling revealed that the mouth region of happy faces was also visually salient by 150 ms following stimulus onset. This suggests that analytical or part-based processing of the salient smile occurs early (150-180 ms) and lateralized (left), and is subsequently used as a shortcut to identify the expression of happiness (350-450 ms). This would account for the happy face advantage in behavioral recognition tasks when the smile is visible.

[1]  S. Hofmann,et al.  Electrophysiological correlates of spatial orienting towards angry faces: A source localization study , 2008, Neuropsychologia.

[2]  Jonathan R. Folstein,et al.  After the P3: late executive processes in stimulus categorization. , 2011, Psychophysiology.

[3]  Francesca Benuzzi,et al.  Fear and happiness in the eyes: An intra-cerebral event-related potential study from the human amygdala , 2012, Neuropsychologia.

[4]  Garrison W. Cottrell,et al.  Transmitting and Decoding Facial Expressions , 2005, Psychological science.

[5]  J. Polich.  Neuropsychology of P300 , 2011 .

[6]  D. Lundqvist,et al.  Facial expressions of emotion (KDEF): Identification under different display-duration conditions , 2008, Behavior research methods.

[7]  Jonas K. Olofsson,et al.  Affective picture processing: An integrative review of ERP findings , 2008, Biological Psychology.

[8]  J. Tanaka,et al.  The NimStim set of facial expressions: Judgments from untrained research participants , 2009, Psychiatry Research.

[9]  L. Nummenmaa,et al.  A smile biases the recognition of eye expressions: configural projection from a salient mouth. , 2013, Quarterly journal of experimental psychology.

[10]  Genevieve McArthur,et al.  Switching associations between facial identity and emotional expression: A behavioural and ERP study , 2010, NeuroImage.

[11]  A. Mizuno,et al.  A change of the leading player in flow Visualization technique , 2006, J. Vis..

[12]  Jennifer J. Richler,et al.  Meanings, Mechanisms, and Measures of Holistic Processing , 2012, Front. Psychology.

[13]  C. Jacques,et al.  Misaligning face halves increases and delays the N170 specifically for upright faces: Implications for the nature of early face representations , 2010, Brain Research.

[14]  Wenfeng Feng,et al.  Three stages of facial expression processing: ERP study with rapid serial visual presentation , 2010, NeuroImage.

[15]  G. Hajcak,et al.  Differentiating neural responses to emotional pictures: evidence from temporal-spatial PCA. , 2009, Psychophysiology.

[16]  R. Adolphs.  Recognizing emotion from facial expressions: psychological and neurological mechanisms. , 2002, Behavioral and cognitive neuroscience reviews.

[17]  K. Fujii,et al.  Visualization for the analysis of fluid motion , 2005, J. Vis..

[18]  Werner Sommer,et al.  Does processing of emotional facial expressions depend on intention? Time-resolved evidence from event-related brain potentials , 2012, Biological Psychology.

[19]  Patrik Vuilleumier,et al.  Time course and specificity of event-related potentials to emotional expressions , 2004, Neuroreport.

[20]  L. Nummenmaa,et al.  Detection of emotional faces: salient physical features guide effective visual search. , 2008, Journal of experimental psychology. General.

[21]  C. Jacques,et al.  The initial representation of individual faces in the right occipito-temporal cortex is holistic: electrophysiological evidence from the composite face illusion. , 2009, Journal of vision.

[22]  Michela Balconi,et al.  Face-selective processing and the effect of pleasant and unpleasant emotional expressions on ERP correlates. , 2003, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[23]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  L. Nummenmaa,et al.  Facial expression recognition in peripheral versus central vision: role of the eyes and the mouth , 2014, Psychological research.

[25]  R. Gur,et al.  Differences in facial expressions of four universal emotions , 2004, Psychiatry Research.

[26]  Jari K. Hietanen,et al.  Differential early ERPs to fearful versus neutral facial expressions: A response to the salience of the eyes? , 2008, Biological Psychology.

[27]  W. Ziegler.  The Oxford Handbook Of Event Related Potential Components , 2016 .

[28]  Manfred Herrmann,et al.  Time course of implicit processing and explicit processing of emotional faces and emotional words , 2011, Biological Psychology.

[29]  J. Tanaka,et al.  Mixed emotions: Holistic and analytic perception of facial expressions , 2010, Cognition & emotion.

[30]  Jari K. Hietanen,et al.  Differential electrocortical responses to increasing intensities of fearful and happy emotional expressions , 2007, Brain Research.

[31]  Annmarie MacNamara,et al.  ERPs and the Study of Emotion , 2011 .

[32]  Christoph M. Michel,et al.  Two electrophysiological stages of spatial orienting towards fearful faces: early temporo-parietal activation preceding gain control in extrastriate visual cortex , 2005, NeuroImage.

[33]  Mathias Weymar,et al.  The face is more than its parts — Brain dynamics of enhanced spatial attention to schematic threat , 2011, NeuroImage.

[34]  Stefanie I. Becker,et al.  Of toothy grins and angry snarls--open mouth displays contribute to efficiency gains in search for emotional faces. , 2012, Journal of vision.

[35]  Michela Balconi,et al.  Consciousness and emotion: ERP modulation and attentive vs. pre-attentive elaboration of emotional facial expressions by backward masking , 2009 .

[36]  F. Mast,et al.  Featural and configural face processing strategies: evidence from a functional magnetic resonance imaging study , 2008, Neuroreport.

[37]  Werner Sommer,et al.  On the automaticity of emotion processing in words and faces: Event-related brain potentials evidence from a superficial task , 2011, Brain and Cognition.

[38]  M. Junghöfer,et al.  The facilitated processing of threatening faces: an ERP analysis. , 2004, Emotion.

[39]  K. Mogg,et al.  Attentional selectivity for emotional faces: evidence from human electrophysiology. , 2009, Psychophysiology.

[40]  T. Allison,et al.  Electrophysiological Studies of Face Perception in Humans , 1996, Journal of Cognitive Neuroscience.

[41]  C. Jacques,et al.  The N170 : understanding the time-course of face perception in the human brain , 2011 .

[42]  D. Fiset,et al.  The eyes are not the window to basic emotions , 2012, Neuropsychologia.

[43]  Juha Silvanto,et al.  Processing of featural and configural aspects of faces is lateralized in dorsolateral prefrontal cortex: A TMS study , 2013, NeuroImage.

[44]  L. Nummenmaa,et al.  Perceptual, categorical, and affective processing of ambiguous smiling facial expressions , 2012, Cognition.

[45]  Margot J. Taylor,et al.  Early processing of the six basic facial emotional expressions. , 2003, Brain research. Cognitive brain research.

[46]  C. Nelson,et al.  Featural and Configural Face Processing in Adults and Infants: A Behavioral and Electrophysiological Investigation , 2006, Perception.

[47]  Masahiro Takei,et al.  Human resource development and visualization , 2009, J. Vis..

[48]  David Beltrán,et al.  When does the brain distinguish between genuine and ambiguous smiles? An ERP study , 2013, Brain and Cognition.

[49]  Le Song,et al.  The ‘when’ and ‘where’ of perceiving signals of threat versus non-threat , 2006, NeuroImage.

[50]  Chad R. Mortensen,et al.  Journal of Experimental Psychology: General Single-and Multiple-target Visual Search Tasks , 2022 .

[51]  Bruno Rossion,et al.  Hemisphere-dependent holistic processing of familiar faces , 2011, Brain and Cognition.

[52]  W. Sommer,et al.  Emotion Effects on the N170: A Question of Reference? , 2012, Brain Topography.

[53]  M. G. Calvo,et al.  Visual search of emotional faces: The role of affective content and featural distinctiveness , 2009 .

[54]  Garrison W. Cottrell,et al.  Transmitting and decoding facial expressions of emotion , 2004 .

[55]  M. G. Calvo,et al.  Recognition advantage of happy faces: Tracing the neurocognitive processes , 2013, Neuropsychologia.

[56]  A. Young,et al.  Configural information in facial expression perception. , 2000, Journal of experimental psychology. Human perception and performance.

[57]  Philippe G. Schyns,et al.  Efficient bubbles for visual categorization tasks , 2011, Vision Research.

[58]  J. Russell,et al.  Universality Revisited , 2013 .

[59]  Anna Weinberg,et al.  Electrocortical responses to NIMSTIM facial expressions of emotion. , 2013, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[60]  C. Junqué,et al.  Sex differences in the human olfactory system , 2006, Brain Research.

[61]  M. Herrmann,et al.  Early and late temporo-spatial effects of contextual interference during perception of facial affect. , 2009, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[62]  M. Pell,et al.  Facial expression decoding as a function of emotional meaning status: ERP evidence , 2009, Neuroreport.

[63]  B. Rossion.  The composite face illusion: A whole window into our understanding of holistic face perception , 2013 .

[64]  Wioleta Walentowska,et al.  Attention modulates emotional expression processing. , 2011, Psychophysiology.

[65]  J. Hietanen,et al.  Is there more in a happy face than just a big smile? , 2007 .

[66]  W. Sommer,et al.  Emotions in word and face processing: Early and late cortical responses , 2009, Brain and Cognition.

[67]  M. Coltheart,et al.  Photographs of facial expression: Accuracy, response times, and ratings of intensity , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[68]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[69]  H. Bülthoff,et al.  The contribution of different facial regions to the recognition of conversational expressions. , 2008, Journal of vision.

[70]  D. Maurer,et al.  Neural correlates of processing facial identity based on features versus their spacing , 2007, Neuropsychologia.