Analysis of convolutional neural networks reveals the computational properties essential for subcortical processing of facial expression

Perception of facial expression is crucial in the social life of primates. This visual information is processed along the ventral cortical pathway and the subcortical pathway. Processing of face information in the subcortical pathway is inaccurate, but the architectural and physiological properties that are responsible remain unclear. We analyzed the performance of convolutional neural networks incorporating three prominent properties of this pathway: a shallow layer architecture, concentric receptive fields at the first processing stage, and a greater degree of spatial pooling. The neural networks designed in this way could be trained to classify seven facial expressions with a correct rate of 51% (chance level, 14%). This modest performance was gradually improved by replacing the three properties, one-by-one, two at a time, or all three simultaneously, with the corresponding features in the cortical pathway. Some processing units in the final layer were sensitive to spatial frequencies (SFs) in the retina-based coordinate, whereas others were sensitive to object-based SFs, similar to neurons in the amygdala. Replacement of any one of these properties affected the SF coordinate of units. All three properties constrain the accuracy of facial expression information in the subcortical pathway, and are essential for determining the coordinate of SF representation.

[1]  I. Fujita,et al.  Convolutional neural networks reveal differences in action units of facial expressions between face image databases developed in different countries , 2022, Frontiers in Psychology.

[2]  M. Tamietto,et al.  A deep neural network model of the primate superior colliculus for emotion recognition , 2022, Philosophical Transactions of the Royal Society B.

[3]  Mariann M. Kiss,et al.  Subcortical Cognition: The Fruit Below the Rind. , 2022, Annual review of neuroscience.

[4]  I. Fujita,et al.  Rapid processing of threatening faces in the amygdala of nonhuman primates: subcortical inputs and dual roles , 2022, Cerebral cortex.

[5]  Jose Ameijeiras-Alonso,et al.  multimode: An R Package for Mode Assessment , 2018, J. Stat. Softw..

[6]  P. Rivas,et al.  A Review of Convolutional Neural Networks and Gabor Filters in Object Recognition , 2020, 2020 International Conference on Computational Science and Computational Intelligence (CSCI).

[7]  M. Bickford,et al.  The Second Visual System of The Tree Shrew , 2019, The Journal of comparative neurology.

[8]  Giorgio Vallortigara,et al.  Cortical route for facelike pattern processing in human newborns , 2019, Proceedings of the National Academy of Sciences.

[9]  Robert L. Whitwell,et al.  Affective blindsight in the absence of input from face processing regions in occipital-temporal cortex , 2017, Neuropsychologia.

[10]  A. Pegna,et al.  Affective blindsight relies on low spatial frequencies , 2017, Neuropsychologia.

[11]  Ziad M. Hafed,et al.  The Foveal Visual Representation of the Primate Superior Colliculus , 2019, Current Biology.

[12]  G. Rizzolatti,et al.  Pathways for smiling, disgust and fear recognition in blindsight patients , 2017, Neuropsychologia.

[13]  S. Yoshikawa,et al.  DEVELOPMENT AND VALIDATION OF THE KOKORO RESEARCH CENTER (KRC) FACIAL EXPRESSION DATABASE , 2019, PSYCHOLOGIA.

[14]  Naokazu Goda,et al.  Neural mechanisms of material perception: quest on Shitsukan , 2018, Neuroscience.

[15]  Chih-Yang Chen,et al.  Spatial frequency sensitivity in macaque midbrain , 2018, Nature Communications.

[16]  Alberto Rodríguez-Casal,et al.  Mode testing, critical bandwidth and excess mass , 2016, TEST.

[17]  B. Strange,et al.  A fast pathway for fear in human amygdala , 2016, Nature Neuroscience.

[18]  W. Freiwald,et al.  Face Processing Systems: From Neurons to Real-World Social Perception. , 2016, Annual review of neuroscience.

[19]  Peter Janssen,et al.  Binocular depth processing in the ventral visual pathway , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[20]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[21]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[23]  Galit Yovel,et al.  A Revised Neural Framework for Face Processing. , 2015, Annual review of vision science.

[24]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[25]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[26]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[27]  Kenta Oono,et al.  Chainer : a Next-Generation Open Source Framework for Deep Learning , 2015 .

[28]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[29]  Jumpei Matsumoto,et al.  Neuronal responses to face-like and facial stimuli in the monkey superior colliculus , 2014, Front. Behav. Neurosci..

[30]  Jumpei Matsumoto,et al.  Pulvinar neurons reveal neurobiological evidence of past selection for rapid detection of snakes , 2013, Proceedings of the National Academy of Sciences.

[31]  S. Kitazawa,et al.  Facilitation of face recognition through the retino-tectal pathway , 2013, Neuropsychologia.

[32]  Maurizio Codispoti,et al.  Spatial frequencies and emotional perception , 2013, Reviews in the neurosciences.

[33]  Dwight J. Kravitz,et al.  The ventral visual pathway: an expanded neural framework for the processing of object quality , 2013, Trends in Cognitive Sciences.

[34]  Jumpei Matsumoto,et al.  Neuronal responses to face‐like stimuli in the monkey pulvinar , 2013, The European journal of neuroscience.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  C. Pack,et al.  Spatiotemporal structure of visual receptive fields in macaque superior colliculus. , 2012, Journal of neurophysiology.

[37]  Bevil R. Conway,et al.  Toward a Unified Theory of Visual Area V4 , 2012, Neuron.

[38]  Eero P. Simoncelli,et al.  Metamers of the ventral stream , 2011, Nature Neuroscience.

[39]  Ichiro Fujita,et al.  Reference Frames for Spatial Frequency in Face Representation Differ in the Temporal Visual Cortex and Amygdala , 2011, The Journal of Neuroscience.

[40]  Skyler T. Hawk,et al.  Presentation and validation of the Radboud Faces Database , 2010 .

[41]  Bevil R. Conway,et al.  Advances in Color Science: From Retina to Behavior , 2010, The Journal of Neuroscience.

[42]  L. Pessoa,et al.  Emotion processing and the amygdala: from a 'low road' to 'many roads' of evaluating biological significance , 2010, Nature Reviews Neuroscience.

[43]  M. Tamietto,et al.  Neural bases of the non-conscious perception of emotional signals , 2010, Nature Reviews Neuroscience.

[44]  Y. Chino,et al.  Receptive‐field properties of V1 and V2 neurons in mice and macaque monkeys , 2010, The Journal of comparative neurology.

[45]  Giuliano Geminiani,et al.  Unseen facial and bodily expressions trigger fast emotional reactions , 2009, Proceedings of the National Academy of Sciences.

[46]  S. Treue,et al.  Diverting Attention Suppresses Human Amygdala Responses to Faces , 2009, NeuroImage.

[47]  Robert A. Marino,et al.  Spatial relationships of visuomotor transformations in the superior colliculus map. , 2008, Journal of neurophysiology.

[48]  Doris Y. Tsao,et al.  Mechanisms of face perception. , 2008, Annual review of neuroscience.

[49]  V. Goffaux,et al.  Newborns’ face recognition is based on spatial frequencies below 0.5 cycles per degree , 2008, Cognition.

[50]  Anitha Pasupathy,et al.  Transformation of shape information in the ventral pathway , 2007, Current Opinion in Neurobiology.

[51]  Mark H. Johnson.  Subcortical face processing , 2005, Nature Reviews Neuroscience.

[52]  Asaid Khateb,et al.  Discriminating emotional faces without primary visual cortices involves the right amygdala , 2005, Nature Neuroscience.

[53]  E. Rolls,et al.  Size and contrast have only small effects on the responses to faces of neurons in the cortex of the superior temporal sulcus of the monkey , 2004, Experimental Brain Research.

[54]  R. Dolan,et al.  Distinct spatial frequency sensitivities for processing faces and emotional expressions , 2003, Nature Neuroscience.

[55]  G. Cottrell,et al.  EMPATH: A Neural Network that Categorizes Facial Expressions , 2002, Journal of Cognitive Neuroscience.

[56]  F. Simion,et al.  Face preference at birth: the role of an orienting mechanism , 2001 .

[57]  L. Weiskrantz,et al.  Differential extrageniculostriate and amygdala responses to presentation of emotional faces in a cortically blind field. , 2001, Brain : a journal of neurology.

[58]  J. Haxby,et al.  The distributed human neural system for face perception , 2000, Trends in Cognitive Sciences.

[59]  L Weiskrantz,et al.  Non-conscious recognition of affect in the absence of striate cortex. , 1999, Neuroreport.

[60]  R. Dolan,et al.  A subcortical pathway to the right amygdala mediating "unseen" fear. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[61]  A. Leventhal,et al.  Signal timing across the macaque visual system. , 1998, Journal of neurophysiology.

[62]  M T Wallace,et al.  Visual response properties and visuotopic representation in the newborn monkey superior colliculus. , 1997, Journal of neurophysiology.

[63]  Minami Ito,et al.  Size and position invariance of neuronal responses in monkey inferotemporal cortex. , 1995, Journal of neurophysiology.

[64]  Minami Ito,et al.  Columns for visual features of objects in monkey inferotemporal cortex , 1992, Nature.

[65]  A. J. Mistlin,et al.  Visual neurones responsive to faces , 1987, Trends in Neurosciences.

[66]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[67]  R. Desimone,et al.  Stimulus-selective properties of inferior temporal neurons in the macaque , 1984, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[68]  Leslie G. Ungerleider.  Two cortical visual systems , 1982 .

[69]  D. B. Bender,et al.  Retinotopic organization of macaque pulvinar. , 1981, Journal of neurophysiology.

[70]  Velma Dobson,et al.  Visual acuity in human infants: A review and comparison of behavioral and electrophysiological studies , 1978, Vision Research.

[71]  B. V. Updyke,et al.  Characteristics of unit responses in superior colliculus of the Cebus monkey. , 1974, Journal of neurophysiology.

[72]  JANETTE ATKINSON,et al.  Acuity and contrast sensivity of infant vision , 1974, Nature.

[73]  R. Wurtz,et al.  Activity of superior colliculus in behaving monkey. I. Visual receptive fields of single neurons. , 1972, Journal of neurophysiology.

[74]  M. Cynader,et al.  Receptive-field organization of monkey superior colliculus. , 1972, Journal of neurophysiology.