A temporal hierarchy of object processing in human visual cortex

Our brain builds increasingly sophisticated representations along the ventral visual pathway to support object recognition, but how these representations unfold over time is poorly understood. Here we characterized time-varying representations of faces, places, and objects throughout the pathway using human intracranial electroencephalography. For ∼100 ms after an initial feedforward sweep, representations at all stages evolved to be less driven by low-order features and more categorical. Low-level areas like V1 showed unexpected, late-emerging tolerance to image size, and late but not early responses of high-level occipitotemporal areas best matched their fMRI responses. Besides aligned, simultaneous representational changes, we found a trial-by-trial association between concurrent response patterns across stages. Our results suggest fast, multi-areal recurrent processing builds upon initial feedforward processing to generate more sophisticated object representations.

[1]  C. Harvey,et al.  The structures and functions of correlations in neural population codes , 2022, Nature Reviews Neuroscience.

[2]  M. Fox,et al.  Causal mapping of human brain function , 2022, Nature Reviews Neuroscience.

[3]  G. Roig,et al.  The spatiotemporal neural dynamics of object location representations in the human brain , 2022, Nature Human Behaviour.

[4]  Daniel L. K. Yamins,et al.  Recurrent Connections in the Primate Ventral Visual Stream Mediate a Tradeoff Between Task Performance and Network Size During Core Object Recognition , 2021, bioRxiv.

[5]  N. P. Bichot,et al.  The cortical connectome of primate lateral prefrontal cortex , 2021, Neuron.

[6]  N. Kanwisher,et al.  Computational models of category-selective brain regions enable high-throughput tests of selectivity , 2021, Nature Communications.

[7]  Yaoda Xu,et al.  Limits to visual representational correspondence between convolutional neural networks and the human brain , 2021, Nature Communications.

[8]  Shuo Wang,et al.  Encoding of facial features by single neurons in the human amygdala and hippocampus , 2020, Communications Biology.

[9]  Kohitij Kar,et al.  Fast recurrent processing via ventral prefrontal cortex is needed by the primate ventral stream for robust core visual object recognition , 2020, bioRxiv.

[10]  Mason McGill,et al.  A map of object space in primate inferotemporal cortex , 2020, Nature.

[11]  Rui Xu,et al.  The role of prefrontal cortex in the control of feature attention in area V4 , 2019, Nature Communications.

[12]  Nikolaus Kriegeskorte,et al.  Recurrence is required to capture the representational dynamics of the human visual system , 2019, Proceedings of the National Academy of Sciences.

[13]  Elias B. Issa,et al.  Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior , 2018, Nature Neuroscience.

[14]  James J DiCarlo,et al.  Neural population control via deep image synthesis , 2018, Science.

[15]  Talia Konkle,et al.  Mid-level visual features underlie the high-level categorical organization of the ventral stream , 2018, Proceedings of the National Academy of Sciences.

[16]  James J. DiCarlo,et al.  Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior , 2018, Nature Neuroscience.

[17]  Dimitrios Pantazis,et al.  Ultra-Rapid serial visual presentation reveals dynamics of feedforward and feedback processes in the ventral visual pathway , 2018, bioRxiv.

[18]  Josef Parvizi,et al.  Promises and limitations of human intracranial electroencephalography , 2018, Nature Neuroscience.

[19]  James J DiCarlo,et al.  Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks , 2018, The Journal of Neuroscience.

[20]  David Cox,et al.  Recurrent computations for visual pattern completion , 2017, Proceedings of the National Academy of Sciences.

[21]  Eric T. Shea-Brown,et al.  Dynamic representation of partially occluded objects in primate prefrontal and visual cortex , 2017, eLife.

[22]  Jesper Andersson,et al.  A multi-modal parcellation of human cerebral cortex , 2016, Nature.

[23]  Josef Parvizi,et al.  Corresponding ECoG and fMRI category-selective signals in human ventral temporal cortex , 2016, Neuropsychologia.

[24]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[25]  John A. Pyles,et al.  Dynamic Encoding of Face Information in the Human Fusiform Gyrus , 2014, Nature Communications.

[26]  Joseph R. Madsen,et al.  Spatiotemporal Dynamics Underlying Object Completion in Human Ventral Visual Cortex , 2014, Neuron.

[27]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[28]  Hualou Liang,et al.  Incremental Integration of Global Contours through Interplay between Visual Cortical Areas , 2014, Neuron.

[29]  Nikolaus Kriegeskorte,et al.  What's there, distinctly, when and where? , 2014, Nature Neuroscience.

[30]  Radoslaw Martin Cichy,et al.  Resolving human object recognition in space and time , 2014, Nature Neuroscience.

[31]  Rajesh P. N. Rao,et al.  Broadband changes in the cortical surface potential track activation of functionally diverse neuronal populations , 2014, NeuroImage.

[32]  Dwight J. Kravitz,et al.  The ventral visual pathway: an expanded neural framework for the processing of object quality , 2013, Trends in Cognitive Sciences.

[33]  K. Grill-Spector,et al.  Neural representations of faces and limbs neighbor in human high-level visual cortex: evidence for a new organization principle , 2011, Psychological research.

[34]  A. Oliva,et al.  A Real-World Size Organization of Object Responses in Occipitotemporal Cortex , 2012, Neuron.

[35]  Joseph R. Madsen,et al.  Individualized localization and cortical surface-based registration of intracranial electrodes , 2012, NeuroImage.

[36]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[37]  J. Maunsell,et al.  Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex , 2011, PLoS biology.

[38]  M. Koivisto,et al.  Recurrent Processing in V1/V2 Contributes to Categorization of Natural Scenes , 2011, The Journal of Neuroscience.

[39]  Jia Liu,et al.  Mapping Informative Clusters in a Hierarchial Framework of fMRI Multivariate Analysis , 2010, PloS one.

[40]  Nicole C. Rust,et al.  Selectivity and Tolerance (“Invariance”) Both Increase as Visual Information Propagates from Cortical Area V4 to IT , 2010, The Journal of Neuroscience.

[41]  Greg O. Horne,et al.  Controlling low-level image properties: The SHINE toolbox , 2010, Behavior research methods.

[42]  Jeremy R. Manning,et al.  Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans , 2009, The Journal of Neuroscience.

[43]  G. Kreiman,et al.  Timing, Timing, Timing: Fast Decoding of Object Information from Intracranial Field Potentials in Human Visual Cortex , 2009, Neuron.

[44]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[45]  Shimon Ullman,et al.  Image interpretation by a single bottom-up top-down cycle , 2008, Proceedings of the National Academy of Sciences.

[46]  N. Kanwisher,et al.  Interpreting fMRI data: maps, modules and dimensions , 2008, Nature Reviews Neuroscience.

[47]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[48]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[49]  K. Grill-Spector,et al.  The human visual cortex. , 2004, Annual review of neuroscience.

[50]  N. Kanwisher,et al.  Stages of processing in face perception: an MEG study , 2002, Nature Neuroscience.

[51]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[52]  Á. Pascual-Leone,et al.  Fast Backprojections from the Motion to the Primary Visual Area Necessary for Visual Awareness , 2001, Science.

[53]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[54]  S Edelman,et al.  Representation is representation of similarities , 1996, Behavioral and Brain Sciences.

[55]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[56]  Denis Fize,et al.  Speed of processing in the human visual system , 1996, Nature.

[57]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.