The representational dynamics of task and object processing in humans

Despite the importance of an observer’s goals in determining how a visual object is categorized, surprisingly little is known about how humans process the task context in which objects occur and how it may interact with the processing of objects. Using magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI) and multivariate techniques, we studied the spatial and temporal dynamics of task and object processing. Our results reveal a sequence of separate but overlapping task-related processes spread across frontoparietal and occipitotemporal cortex. Task exhibited late effects on object processing by selectively enhancing task-relevant object features, with limited impact on the overall pattern of object representations. Combining MEG and fMRI data, we reveal a parallel rise in task-related signals throughout the cerebral cortex, with an increasing dominance of task over object representations from early to higher visual areas. Collectively, our results reveal the complex dynamics underlying task and object representations throughout human cortex.

[1]  Daria Proklova,et al.  Disentangling Representations of Object Shape and Object Category in Human Visual Cortex: The Animate–Inanimate Distinction , 2016, Journal of Cognitive Neuroscience.

[2]  Richard M. Leahy,et al.  Brainstorm: A User-Friendly Application for MEG/EEG Analysis , 2011, Comput. Intell. Neurosci..

[3]  H. Esteky,et al.  Behavioral demand modulates object category representation in the inferior temporal cortex. , 2014, Journal of Neurophysiology.

[4]  Radoslaw Martin Cichy,et al.  Resolving human object recognition in space and time , 2014, Nature Neuroscience.

[5]  David J. Freedman,et al.  Neuronal Mechanisms of Visual Categorization: An Abstract View on Decision Making. , 2016, Annual review of neuroscience.

[6]  Stefan Bode,et al.  Decoding sequential stages of task preparation in the human brain , 2009, NeuroImage.

[7]  Kalanit Grill-Spector,et al.  Task alters category representations in prefrontal but not high-level visual cortex , 2017, NeuroImage.

[8]  David A. Tovar,et al.  Representational dynamics of object vision: the first 1000 ms. , 2013, Journal of vision.

[9]  J. Duncan.  The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour , 2010, Trends in Cognitive Sciences.

[10]  S. Thorpe,et al.  The Time Course of Visual Processing: From Early Perception to Decision-Making , 2001, Journal of Cognitive Neuroscience.

[11]  Dimitrios Pantazis,et al.  Similarity-Based Fusion of MEG and fMRI Reveals Spatio-Temporal Dynamics in Human Cortex During Visual Object Recognition , 2015, bioRxiv.

[12]  Martin N. Hebart,et al.  Human visual and parietal cortex encode visual choices independent of motor plans , 2012, NeuroImage.

[13]  H. P. Op de Beeck,et al.  Dissociations and Associations between Shape and Category Representations in the Two Visual Pathways , 2015, The Journal of Neuroscience.

[14]  V. Lamme,et al.  The time course of natural scene perception with reduced attention. , 2016, Journal of neurophysiology.

[15]  Samuel A. Nastase,et al.  Attention Selectively Reshapes the Geometry of Distributed Semantic Representation , 2016, bioRxiv.

[16]  Dwight J. Kravitz,et al.  High-level visual object representations are constrained by position. , 2010, Cerebral cortex.

[17]  T. Poggio,et al.  Neural mechanisms of object recognition , 2002, Current Opinion in Neurobiology.

[18]  Chris I. Baker,et al.  The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks , 2017, NeuroImage.

[19]  Chris I. Baker,et al.  Deconstructing multivariate decoding for the study of brain function , 2017, NeuroImage.

[20]  Markus Siegel,et al.  Cortical information flow during flexible sensorimotor decisions , 2015, Science.

[21]  Moritz Grosse-Wentrup,et al.  Multisubject Learning for Common Spatial Patterns in Motor-Imagery BCI , 2011, Comput. Intell. Neurosci..

[22]  John Duncan,et al.  Hierarchical coding for sequential task events in the monkey prefrontal cortex , 2008, Proceedings of the National Academy of Sciences.

[23]  Janneke F. M. Jehee,et al.  Attention Improves Encoding of Task-Relevant Features in the Human Visual Cortex , 2011, The Journal of Neuroscience.

[24]  David J. Freedman,et al.  Dynamic population coding of category information in inferior temporal and prefrontal cortex. , 2008, Journal of neurophysiology.

[25]  Martin N. Hebart,et al.  The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data , 2015, Front. Neuroinform..

[26]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[27]  Dwight J. Kravitz,et al.  Task context impacts visual object processing differentially across the cortex , 2014, Proceedings of the National Academy of Sciences.

[28]  Thomas Serre,et al.  A feedforward architecture accounts for rapid categorization , 2007, Proceedings of the National Academy of Sciences.

[29]  Michelle R. Greene,et al.  Visual scenes are categorized by function. , 2016, Journal of experimental psychology. General.

[30]  Jonathan S. Cant,et al.  Feature diagnosticity and task context shape activity in human scene-selective cortex , 2016, NeuroImage.

[31]  Lawrence H Snyder,et al.  Single Neurons in Posterior Parietal Cortex of Monkeys Encode Cognitive Set , 2004, Neuron.

[32]  Janneke F. M. Jehee,et al.  Less Is More: Expectation Sharpens Representations in the Primary Visual Cortex , 2012, Neuron.

[33]  Yaoda Xu,et al.  Goal-Directed Visual Processing Differentially Impacts Human Ventral and Dorsal Visual Representations , 2017, The Journal of Neuroscience.

[34]  Daria Proklova,et al.  MEG sensor patterns reflect perceptual but not categorical similarity of animate and inanimate objects , 2018, NeuroImage.

[35]  Susan G. Wardle,et al.  Decoding Dynamic Brain Patterns from Evoked Responses: A Tutorial on Multivariate Pattern Analysis Applied to Time Series Neuroimaging Data , 2016, Journal of Cognitive Neuroscience.

[36]  Stefania Bracci,et al.  Avoiding illusory effects in representational similarity analysis: What (not) to do with the diagonal , 2017, NeuroImage.

[37]  Graham W. Taylor,et al.  TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE , 2018 .

[38]  E. Pedhazur.  Multiple Regression in Behavioral Research: Explanation and Prediction , 1982 .

[39]  Michael L. Waskom,et al.  Frontoparietal Representations of Task Context Support the Flexible Control of Goal-Directed Cognition , 2014, The Journal of Neuroscience.

[40]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[41]  Alexander G. Huth,et al.  Attention During Natural Vision Warps Semantic Representation Across the Human Brain , 2013, Nature Neuroscience.

[42]  Joel Z. Leibo,et al.  The dynamics of invariant object recognition in the human visual system. , 2014, Journal of neurophysiology.

[43]  N. Sigala,et al.  Dynamic Coding for Cognitive Control in Prefrontal Cortex , 2013, Neuron.

[44]  S. Kastner,et al.  A neural basis for real-world visual search in human occipitotemporal cortex , 2011, Proceedings of the National Academy of Sciences.

[45]  Li Fei-Fei,et al.  Neural mechanisms of rapid natural scene categorization in human visual cortex , 2009, Nature.

[46]  David J. Freedman,et al.  Task Dependence of Visual and Category Representations in Prefrontal and Inferior Temporal Cortices , 2014, The Journal of Neuroscience.

[47]  K. C. Anderson,et al.  Single neurons in prefrontal cortex encode abstract rules , 2001, Nature.

[48]  L. Tyler,et al.  Predicting the Time Course of Individual Objects with MEG , 2014, Cerebral cortex.

[49]  Thomas E. Nichols,et al.  Nonparametric permutation tests for functional neuroimaging: A primer with examples , 2002, Human brain mapping.

[50]  Robert D. McPhee,et al.  Commonality Analysis: A Method for Decomposing Explained Variance in Multiple Regression Analyses. , 1979 .

[51]  David J. Freedman,et al.  A Comparison of Primate Prefrontal and Inferior Temporal Cortices during Visual Categorization , 2003, The Journal of Neuroscience.

[52]  S. Dehaene,et al.  Characterizing the dynamics of mental representations: the temporal generalization method , 2014, Trends in Cognitive Sciences.

[53]  Yi Chen,et al.  Encoding the identity and location of objects in human LOC , 2011, NeuroImage.

[54]  Su Keun Jeong,et al.  Behaviorally Relevant Abstract Object Identity Representation in the Human Parietal Cortex , 2016, The Journal of Neuroscience.

[55]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[56]  H. P. Op de Beeck,et al.  Task Context Overrules Object- and Category-Related Representational Content in the Human Parietal Cortex , 2017, Cerebral cortex.

[57]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[58]  Marcel van Gerven,et al.  MEG-based decoding of the spatiotemporal dynamics of visual category perception , 2013, NeuroImage.

[59]  Jochen Kaiser,et al.  Recurrence of task set-related MEG signal patterns during auditory working memory , 2016, Brain Research.

[60]  Marius V Peelen,et al.  Shape-independent object category responses revealed by MEG and fMRI decoding. , 2016, Journal of neurophysiology.

[61]  S. Kastner,et al.  Two hierarchically organized neural systems for object information in human visual cortex , 2008, Nature Neuroscience.

[62]  Dimitrios Pantazis,et al.  Similarity-based fusion of MEG and fMRI reveals spatio-temporal dynamics in human cortex during visual object recognition , 2015 .

[63]  J. Duncan,et al.  Discrimination of Visual Categories Based on Behavioral Relevance in Widespread Regions of Frontoparietal Cortex , 2015, The Journal of Neuroscience.

[64]  F. D. de Lange,et al.  Prior Expectations Bias Sensory Representations in Visual Cortex , 2013, The Journal of Neuroscience.

[65]  John Duncan,et al.  Multi-voxel coding of stimuli, rules, and responses in human frontoparietal cortex , 2011, NeuroImage.

[66]  S. Edelman,et al.  Differential Processing of Objects under Various Viewing Conditions in the Human Lateral Occipital Complex , 1999, Neuron.

[67]  Thomas A. Carlson,et al.  Emerging Object Representations in the Visual System Predict Reaction Times for Categorization , 2015, PLoS Comput. Biol..

[68]  Dirk Walther,et al.  Category discrimination of early electrophysiological responses reveals the time course of natural scene perception , 2017 .