Evidence that the ventral stream codes the errors used in hierarchical inference and learning

Hierarchical feedforward processing makes object identity explicit at the highest stages of the ventral visual stream. We leveraged this computational goal to study the fine-scale temporal dynamics of neural populations in posterior and anterior inferior temporal cortex (pIT, aIT) during face detection. As expected, we found that a neural spiking preference for natural over distorted face images was rapidly produced, first in pIT and then in aIT. Strikingly, in the next 30 milliseconds of processing, this pattern of selectivity in pIT completely reversed, while selectivity in aIT remained unchanged. Although these dynamics were difficult to explain from a pure feedforward perspective, a model class computing errors through feedback closely matched the observed neural dynamics and parsimoniously explained a range of seemingly disparate IT neural response phenomena. This new perspective augments the standard model of online vision by suggesting that neural signals of states (e.g. likelihood of a face being present) are intermixed with the error signals found in deep hierarchical networks.

[1]  Richard T Born,et al.  Corticocortical Feedback Contributes to Surround Suppression in V1 of the Alert Primate , 2013, The Journal of Neuroscience.

[2]  Nicole C. Rust,et al.  Selectivity and Tolerance (“Invariance”) Both Increase as Visual Information Propagates from Cortical Area V4 to IT , 2010, The Journal of Neuroscience.

[3]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[4]  R. Kiani,et al.  Microstimulation of inferotemporal cortex influences face categorization , 2006, Nature.

[5]  Doris Y. Tsao,et al.  A face feature space in the macaque temporal lobe , 2009, Nature Neuroscience.

[6]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[7]  Doris Y. Tsao,et al.  A Cortical Region Consisting Entirely of Face-Selective Cells , 2006, Science.

[8]  Tai Sing Lee,et al.  Hierarchical Bayesian inference in the visual cortex. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[9]  Elias B. Issa,et al.  Large-Scale, High-Resolution Neurophysiological Maps Underlying fMRI of Macaque Temporal Lobe , 2013, The Journal of Neuroscience.

[10]  J. Movshon,et al.  Linearity and Normalization in Simple Cells of the Macaque Primary Visual Cortex , 1997, The Journal of Neuroscience.

[11]  A. Borst.  Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[12]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[13]  Scott L. Brincat,et al.  Dynamic Shape Synthesis in Posterior Inferotemporal Cortex , 2006, Neuron.

[14]  Doris Y. Tsao,et al.  Comparing face patch systems in macaques and humans , 2008, Proceedings of the National Academy of Sciences.

[15]  Joseph R. Madsen,et al.  Spatiotemporal Dynamics Underlying Object Completion in Human Ventral Visual Cortex , 2014, Neuron.

[16]  J. Nassi,et al.  Corticocortical feedback increases the spatial extent of normalization , 2014, Front. Syst. Neurosci..

[17]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[18]  Daniel L. K. Yamins,et al.  Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition , 2014, PLoS Comput. Biol..

[19]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[20]  Aapo Hyvärinen,et al.  Natural Image Statistics - A Probabilistic Approach to Early Computational Vision , 2009, Computational Imaging and Vision.

[21]  Doris Y. Tsao,et al.  The Code for Facial Identity in the Primate Brain , 2017, Cell.

[22]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[23]  Konrad P. Körding,et al.  Toward an Integration of Deep Learning and Neuroscience , 2016, bioRxiv.

[24]  Doris Y. Tsao,et al.  Intelligent Information Loss: The Coding of Facial Identity, Head Pose, and Non-Face Information in the Macaque Face Patch System , 2015, The Journal of Neuroscience.

[25]  E. Miller,et al.  Experience-dependent sharpening of visual shape selectivity in inferior temporal cortex. , 2005, Cerebral cortex.

[26]  Doris Y. Tsao,et al.  What Makes a Cell Face Selective? The Importance of Contrast , 2012, Neuron.

[27]  Pascal Vincent,et al.  Contractive Auto-Encoders: Explicit Invariance During Feature Extraction , 2011, ICML.

[28]  Richard G. Baraniuk,et al.  A Probabilistic Theory of Deep Learning , 2015, ArXiv.

[29]  Karl J. Friston,et al.  Cortical circuits for perceptual inference , 2009, Neural Networks.

[30]  Ha Hong,et al.  Simple Learned Weighted Sums of Inferior Temporal Neuronal Firing Rates Accurately Predict Human Core Object Recognition Performance , 2015, The Journal of Neuroscience.

[31]  C. Olson,et al.  Image familiarization sharpens response dynamics of neurons in inferotemporal cortex , 2014, Nature Neuroscience.

[32]  David L. Sheinberg,et al.  Effects of Long-Term Visual Experience on Responses of Distinct Classes of Single Units in Inferior Temporal Cortex , 2012, Neuron.

[33]  J. Movshon,et al.  Nature and interaction of signals from the receptive field center and surround in macaque V1 neurons. , 2002, Journal of neurophysiology.

[34]  S. Gerber,et al.  Unsupervised Natural Experience Rapidly Alters Invariant Object Representation in Visual Cortex , 2008 .

[35]  L. Saksida,et al.  Visual perception and memory: a new view of medial temporal lobe function in primates and rodents. , 2007, Annual review of neuroscience.

[36]  Eero P. Simoncelli,et al.  Natural signal statistics and sensory gain control , 2001, Nature Neuroscience.

[37]  Elias B. Issa,et al.  Precedence of the Eye Region in Neural Processing of Faces , 2012, The Journal of Neuroscience.

[38]  Hualou Liang,et al.  Incremental Integration of Global Contours through Interplay between Visual Cortical Areas , 2014, Neuron.

[39]  E. Adelson,et al.  Phenomenal coherence of moving visual patterns , 1982, Nature.

[40]  Konrad P. Kording,et al.  Towards an integration of deep learning and neuroscience , 2016, bioRxiv.

[41]  Caspar M. Schwiedrzik,et al.  High-Level Prediction Signals in a Low-Level Area of the Macaque Face-Processing Hierarchy , 2017, Neuron.

[42]  Wilbert Zarco,et al.  A causal relationship between face-patch activity and face-detection behavior , 2017, eLife.

[43]  Doris Y. Tsao,et al.  Patches with Links: A Unified System for Processing Faces in the Macaque Temporal Lobe , 2008, Science.

[44]  Shimon Ullman,et al.  Image interpretation by a single bottom-up top-down cycle , 2008, Proceedings of the National Academy of Sciences.

[45]  Sebastian,et al.  Pattern Analysis and Synthesis in AttractorNeural NetworksH , 1997 .

[46]  M. Giese,et al.  Norm-based face encoding by single neurons in the monkey inferotemporal cortex , 2006, Nature.

[47]  C. Connor,et al.  Shape representation in area V4: position-specific tuning for boundary conformation. , 2001, Journal of neurophysiology.

[48]  David D. Cox,et al.  High-resolution three-dimensional microelectrode brain mapping using stereo microfocal X-ray imaging. , 2008, Journal of neurophysiology.

[49]  M. Behrmann,et al.  Impact of learning on representation of parts and wholes in monkey inferotemporal cortex , 2002, Nature Neuroscience.

[50]  D. Mumford,et al.  Neural activity in early visual cortex reflects behavioral experience and higher-order perceptual saliency , 2002, Nature Neuroscience.

[51]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[52]  C. Olson,et al.  Statistical learning of visual transitions in monkey inferotemporal cortex , 2011, Proceedings of the National Academy of Sciences.

[53]  Dicarlo James.  Towards large-scale, high resolution maps of object selectivity in inferior temporal cortex , 2010 .

[54]  Doris Y. Tsao,et al.  Functional Compartmentalization and Viewpoint Generalization Within the Macaque Face-Processing System , 2010, Science.

[55]  Kenji Kawano,et al.  Global and fine information coded by single neurons in the temporal visual cortex , 1999, Nature.

[56]  P. Berkes,et al.  Statistically Optimal Perception and Learning: from Behavior to Neural Representations , 2022 .

[57]  Rob Fergus,et al.  Stochastic Pooling for Regularization of Deep Convolutional Neural Networks , 2013, ICLR.

[58]  Doris Y. Tsao,et al.  Anatomical Connections of the Functionally Defined “Face Patches” in the Macaque Monkey , 2016, Neuron.

[59]  S. Yantis,et al.  Visual attention: control, representation, and time course. , 1997, Annual review of psychology.

[60]  Elias B. Issa,et al.  Neurophysiological Organization of the Middle Face Patch in Macaque Inferior Temporal Cortex , 2016, The Journal of Neuroscience.

[61]  R. von der Heydt,et al.  Analysis of the Context Integration Mechanisms Underlying Figure–Ground Organization in the Visual Cortex , 2010, The Journal of Neuroscience.

[62]  O. Schwartz,et al.  Flexible Gating of Contextual Influences in Natural Vision , 2015, Nature Neuroscience.

[63]  Doris Y. Tsao,et al.  The effect of face patch microstimulation on perception of faces and objects , 2017, Nature Neuroscience.

[64]  S. Morad,et al.  Ceramide-orchestrated signalling in cancer cells , 2012, Nature Reviews Cancer.

[65]  Geoffrey E. Hinton,et al.  An Efficient Learning Procedure for Deep Boltzmann Machines , 2012, Neural Computation.