Coding of Visual, Auditory, Rule, and Response Information in the Brain: 10 Years of Multivoxel Pattern Analysis

How is the processing of task information organized in the brain? Many views of brain function emphasize modularity, with different regions specialized for processing different types of information. However, recent accounts also highlight flexibility, pointing especially to the highly consistent pattern of frontoparietal activation across many tasks. Although early insights from functional imaging were based on overall activation levels during different cognitive operations, in the last decade many researchers have used multivoxel pattern analyses to interrogate the representational content of activations, mapping out the brain regions that make particular stimulus, rule, or response distinctions. Here, we drew on 100 searchlight decoding analyses from 57 published papers to characterize the information coded in different brain networks. The outcome was highly structured. Visual, auditory, and motor networks predominantly (but not exclusively) coded visual, auditory, and motor information, respectively. By contrast, the frontoparietal multiple-demand network was characterized by domain generality, coding visual, auditory, motor, and rule information. The contribution of the default mode network and voxels elsewhere was minor. The data suggest a balanced picture of brain organization in which sensory and motor networks are relatively specialized for information in their own domain, whereas a specific frontoparietal network acts as a domain-general “core” with the capacity to code many different aspects of a task.

[1]  G. Egan,et al.  Led into Temptation? Rewarding Brand Logos Bias the Neural Encoding of Incidental Economic Decisions , 2012, PloS one.

[2]  Masataka Watanabe,et al.  Prefrontal unit activity and delayed response: Relation to cue location versus direction of response , 1976, Brain Research.

[3]  Joshua W. Brown,et al.  Rostral–caudal gradients of abstraction revealed by multi-variate pattern analysis of working memory , 2012, NeuroImage.

[4]  Alexander Borst,et al.  How does Nature Program Neuron Types? , 2008, Front. Neurosci..

[5]  Justin L. Vincent,et al.  Distinct brain networks for adaptive and stable task control in humans , 2007, Proceedings of the National Academy of Sciences.

[6]  C. Fiebach,et al.  Predicting errors from reconfiguration patterns in human brain networks , 2012, Proceedings of the National Academy of Sciences.

[7]  J. Duncan An adaptive coding model of neural function in prefrontal cortex , 2001 .

[8]  J. Tanji,et al.  Task-dependent selectivity of movement-related neuronal activity in the primate prefrontal cortex. , 1998, Journal of neurophysiology.

[9]  Tobias Teichert,et al.  The dorsal medial frontal cortex is sensitive to time on task, not response conflict or error likelihood , 2011, NeuroImage.

[10]  Polina Golland,et al.  Coping with confounds in multivoxel pattern analysis: What should we do about reaction time differences? A comment on Todd, Nystrom & Cohen 2013 , 2014, NeuroImage.

[11]  Alejandro Vicente-Grabovetsky,et al.  Multivoxel Patterns Reveal Functionally Differentiated Networks Underlying Auditory Feedback Processing of Speech , 2013, The Journal of Neuroscience.

[12]  Daniel J Mitchell,et al.  Recruitment of the default mode network during a demanding act of executive control , 2015, eLife.

[13]  S. Scott,et al.  Intact But Less Accessible Phonetic Representations in Adults with Dyslexia , 2013, Science.

[14]  R. Passingham,et al.  Reading Hidden Intentions in the Human Brain , 2007, Current Biology.

[15]  Kristina M. Visscher,et al.  A Core System for the Implementation of Task Sets , 2006, Neuron.

[16]  M. Peelen,et al.  Supramodal Representations of Perceived Emotions in the Human Brain , 2010, The Journal of Neuroscience.

[17]  John-Dylan Haynes,et al.  The Neural Representation of Voluntary Task-Set Selection in Dynamic Environments. , 2015, Cerebral cortex.

[18]  M. Azuma,et al.  Properties and distribution of auditory neurons in the dorsolateral prefrontal cortex of the alert monkey , 1984, Brain Research.

[19]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[20]  Martin N. Hebart,et al.  Human visual and parietal cortex encode visual choices independent of motor plans , 2012, NeuroImage.

[21]  S Kornblum,et al.  Dynamics of Single Neuron Activity in Monkey Primary Motor Cortex Related to Sensorimotor Transformation , 1997, The Journal of Neuroscience.

[22]  Kathryn M. McMillan,et al.  N‐back working memory paradigm: A meta‐analysis of normative functional neuroimaging studies , 2005, Human brain mapping.

[23]  Tobias Egner,et al.  Quantify the Modularity of Con fl ict – Control Mechanisms in the Human Brain , 2014 .

[24]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[25]  Stefan Bode,et al.  Similar neural mechanisms for perceptual guesses and free decisions , 2013, NeuroImage.

[26]  Johan D. Carlin,et al.  A Head View-Invariant Representation of Gaze Direction in Anterior Superior Temporal Sulcus , 2011, Current Biology.

[27]  L. Chelazzi,et al.  Neurons in Area V4 of the Macaque Translate Attended Visual Features into Behaviorally Relevant Categories , 2007, Neuron.

[28]  Haynes John-Dylan,et al.  Perceptual learning and decision making in human medial frontal cortex , 2012 .

[29]  Stefan Bode,et al.  Decoding sequential stages of task preparation in the human brain , 2009, NeuroImage.

[30]  Richard Granger,et al.  Categorical Speech Processing in Broca's Area: An fMRI Study Using Multivariate Pattern-Based Analysis , 2012, The Journal of Neuroscience.

[31]  S Kornblum,et al.  Neuronal coding of stimulus-response association rules in the motor cortex. , 1994, Neuroreport.

[32]  Scott A. Huettel,et al.  Within- and cross-participant classifiers reveal different neural coding of information , 2011, NeuroImage.

[33]  C. Summerfield,et al.  An information theoretical approach to prefrontal executive function , 2007, Trends in Cognitive Sciences.

[34]  Thad A. Polk,et al.  Age-Related Neural Dedifferentiation in the Motor System , 2011, PloS one.

[35]  J. Duncan The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour , 2010, Trends in Cognitive Sciences.

[36]  John-Dylan Haynes,et al.  Distributed Representations of Rule Identity and Rule Order in Human Frontal Cortex and Striatum , 2012, The Journal of Neuroscience.

[37]  Nancy Kanwisher,et al.  Broad domain generality in focal regions of frontal and parietal cortex , 2013, Proceedings of the National Academy of Sciences.

[38]  P. Downing,et al.  Mapping brain activation and information during category-specific visual working memory. , 2012, Journal of neurophysiology.

[39]  M. Brass,et al.  Unconscious determinants of free decisions in the human brain , 2008, Nature Neuroscience.

[40]  E. Miller,et al.  Neural Activity in the Primate Prefrontal Cortex during Associative Learning , 1998, Neuron.

[41]  Johan D. Carlin,et al.  Direction-Sensitive Codes for Observed Head Turns in Human Superior Temporal Sulcus , 2011, Cerebral cortex.

[42]  Bruno L. Giordano,et al.  Abstract encoding of auditory objects in cortical activity patterns. , 2013, Cerebral cortex.

[43]  Jac Billington,et al.  Cortical responses to congruent and incongruent stereo cues for objects on a collision path with the observer , 2013, Displays.

[44]  G. H. Freeman,et al.  Note on an exact treatment of contingency, goodness of fit and other problems of significance. , 1951, Biometrika.

[45]  G. Glover,et al.  Dissociable Intrinsic Connectivity Networks for Salience Processing and Executive Control , 2007, The Journal of Neuroscience.

[46]  Soon Chun Siong,et al.  Pre-existing brain states predict risky choices , 2014, NeuroImage.

[47]  Daeyeol Lee,et al.  Ubiquity and Specificity of Reinforcement Signals throughout the Human Brain , 2011, Neuron.

[48]  Timothy O. Laumann,et al.  Functional Network Organization of the Human Brain , 2011, Neuron.

[49]  John Duncan,et al.  Hierarchical coding for sequential task events in the monkey prefrontal cortex , 2008, Proceedings of the National Academy of Sciences.

[50]  Dwight J. Kravitz,et al.  Task context impacts visual object processing differentially across the cortex , 2014, Proceedings of the National Academy of Sciences.

[51]  Jörg Bahlmann,et al.  Predicting vocal emotion expressions from the human brain , 2013, Human brain mapping.

[52]  Daniel C. McNamee,et al.  Category-dependent and category-independent goal-value codes in human ventromedial prefrontal cortex , 2013, Nature Neuroscience.

[53]  L. Nyberg,et al.  Common fronto-parietal activity in attention, memory, and consciousness: Shared demands on integration? , 2005, Consciousness and Cognition.

[54]  David Badre,et al.  Functional Magnetic Resonance Imaging Evidence for a Hierarchical Organization of the Prefrontal Cortex , 2007, Journal of Cognitive Neuroscience.

[55]  J. Obleser,et al.  Dissociable neural imprints of perception and grammar in auditory functional imaging , 2012, Human brain mapping.

[56]  A. Caramazza,et al.  Conceptual Object Representations in Human Anterior Temporal Cortex , 2012, The Journal of Neuroscience.

[57]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[58]  Thomas H. B. FitzGerald,et al.  Reward-Related Activity in Ventral Striatum Is Action Contingent and Modulated by Behavioral Relevance , 2014, The Journal of Neuroscience.

[59]  S Kornblum,et al.  Neuronal correlates of sensorimotor association in stimulus-response compatibility. , 1997, Journal of experimental psychology. Human perception and performance.

[60]  Robert Oostenveld,et al.  Modality-independent decoding of semantic information from the human brain. , 2014, Cerebral cortex.

[61]  Adrian M. Owen,et al.  Fractionating Human Intelligence , 2012, Neuron.

[62]  John-Dylan Haynes,et al.  Diagnosing different binge‐eating disorders based on reward‐related brain activation patterns , 2012, Human brain mapping.

[63]  J. Duncan,et al.  Discrimination of Visual Categories Based on Behavioral Relevance in Widespread Regions of Frontoparietal Cortex , 2015, The Journal of Neuroscience.

[64]  Michael W. L. Chee,et al.  Preparatory patterns of neural activity predict visual category search speed , 2013, NeuroImage.

[65]  Maurizio Corbetta,et al.  The human brain is intrinsically organized into dynamic, anticorrelated functional networks. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[66]  J. Duncan EPS Mid-Career Award 2004: Brain mechanisms of attention , 2006, Quarterly journal of experimental psychology.

[67]  John-Dylan Haynes,et al.  Human anterior prefrontal cortex encodes the ‘what’ and ‘when’ of future intentions , 2012, NeuroImage.

[68]  J. Duncan,et al.  Adaptive Coding of Task-Relevant Information in Human Frontoparietal Cortex , 2011, The Journal of Neuroscience.

[69]  J. Duncan,et al.  Common regions of the human frontal lobe recruited by diverse cognitive demands , 2000, Trends in Neurosciences.

[70]  Stephen D. Mayhew,et al.  Dissociable circuits for visual shape learning in the young and aging human brain , 2013, Front. Hum. Neurosci..

[71]  Jaron T Colas,et al.  Pre‐existing brain states predict aesthetic judgments , 2014, Human brain mapping.

[72]  Stefan Pollmann,et al.  The right temporo-parietal junction contributes to visual feature binding , 2014, NeuroImage.

[73]  D. Schacter,et al.  The Brain's Default Network , 2008, Annals of the New York Academy of Sciences.

[74]  Sheng Li,et al.  Task‐dependent uncertainty modulation of perceptual decisions in the human brain , 2012, The European journal of neuroscience.

[75]  L. Tyler,et al.  Object-Specific Semantic Coding in Human Perirhinal Cortex , 2014, The Journal of Neuroscience.

[76]  J. Haynes,et al.  Decoding Successive Computational Stages of Saliency Processing , 2011, Current Biology.

[77]  G L Shulman,et al.  INAUGURAL ARTICLE by a Recently Elected Academy Member:A default mode of brain function , 2001 .

[78]  Anina N. Rich,et al.  Flexible Coding of Task Rules in Frontoparietal Cortex: An Adaptive System for Flexible Cognitive Control , 2015, Journal of Cognitive Neuroscience.

[79]  Daryl E. Wilson,et al.  Control of Spatial and Feature-Based Attention in Frontoparietal Cortex , 2010, The Journal of Neuroscience.

[80]  Jonas T. Kaplan,et al.  Multivariate pattern analysis reveals common neural patterns across individuals during touch observation , 2012, NeuroImage.

[81]  R. O’Reilly The What and How of prefrontal cortical organization , 2010, Trends in Neurosciences.

[82]  John Duncan,et al.  Dynamic Construction of a Coherent Attentional State in a Prefrontal Cell Population , 2013, Neuron.

[83]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[84]  Anina N. Rich,et al.  Attention enhances multi-voxel representation of novel objects in frontal, parietal and visual cortices , 2015, NeuroImage.

[85]  David J. Freedman,et al.  Experience-dependent representation of visual categories in parietal cortex , 2006, Nature.

[86]  S. Dehaene,et al.  Towards a cognitive neuroscience of consciousness: basic evidence and a workspace framework , 2001, Cognition.

[87]  M. Corbetta,et al.  Control of goal-directed and stimulus-driven attention in the brain , 2002, Nature Reviews Neuroscience.

[88]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[89]  Ione Fine,et al.  Auditory motion processing after early blindness. , 2014, Journal of vision.

[90]  N. Sigala,et al.  Dynamic Coding for Cognitive Control in Prefrontal Cortex , 2013, Neuron.

[91]  Jonathan D. Power,et al.  Control-related systems in the human brain , 2013, Current Opinion in Neurobiology.

[92]  K. C. Anderson,et al.  Single neurons in prefrontal cortex encode abstract rules , 2001, Nature.

[93]  Ben M. Crittenden,et al.  Task Difficulty Manipulation Reveals Multiple Demand Activity but no Frontal Lobe Hierarchy , 2012, Cerebral cortex.

[94]  S. Gilbert,et al.  Evaluative vs. trait representation in intergroup social judgments: Distinct roles of anterior temporal lobe and prefrontal cortex , 2012, Neuropsychologia.

[95]  W. Singer,et al.  Auditory motion direction encoding in auditory cortex and high‐level visual cortex , 2012, Human brain mapping.

[96]  P. Goldman-Rakic The prefrontal landscape: implications of functional architecture for understanding human mentation and the central executive. , 1996, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[97]  John Duncan,et al.  Multi-voxel coding of stimuli, rules, and responses in human frontoparietal cortex , 2011, NeuroImage.

[98]  David J. Freedman,et al.  Categorical representation of visual stimuli in the primate prefrontal cortex. , 2001, Science.

[99]  Johan D. Carlin,et al.  Choosing the Rules: Distinct and Overlapping Frontoparietal Representations of Task Rules for Perceptual Decisions , 2013, The Journal of Neuroscience.

[100]  Stephen D. Mayhew,et al.  Learning Shapes the Representation of Visual Categories in the Aging Human Brain , 2010, Journal of Cognitive Neuroscience.

[101]  Lizabeth M Romanski,et al.  Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. , 2007, Cerebral cortex.

[102]  J. Haynes,et al.  Perceptual Learning and Decision-Making in Human Medial Frontal Cortex , 2011, Neuron.

[103]  Soon Chun Siong,et al.  The neural encoding of guesses in the human brain , 2012, NeuroImage.

[104]  Tyrone D. Cannon,et al.  Predicting risky choices from brain activity patterns , 2014, Proceedings of the National Academy of Sciences.

[105]  Kimberly L. Ray,et al.  Meta-analytic evidence for a superordinate cognitive control network subserving diverse executive functions , 2012, Cognitive, affective & behavioral neuroscience.

[106]  S C Rao,et al.  Integration of what and where in the primate prefrontal cortex. , 1997, Science.

[107]  S. Wise,et al.  Rule-dependent neuronal activity in the prefrontal cortex , 1999, Experimental Brain Research.

[108]  John-Dylan Haynes,et al.  Decoding the Contents of Visual Short-Term Memory from Human Visual and Parietal Cortex , 2012, The Journal of Neuroscience.

[109]  Jonathan D. Cohen,et al.  Confounds in multivariate pattern analysis: Theory and rule representation case study , 2013, NeuroImage.

[110]  S. Gilbert Decoding the Content of Delayed Intentions , 2011, The Journal of Neuroscience.

[111]  M. Botvinick Hierarchical models of behavior and prefrontal function , 2008, Trends in Cognitive Sciences.

[112]  John-Dylan Haynes,et al.  Compositionality of rule representations in human prefrontal cortex. , 2012, Cerebral cortex.

[113]  R. Cusack,et al.  Stimulus-specific suppression preserves information in auditory short-term memory , 2011, Proceedings of the National Academy of Sciences.

[114]  M. Bangert,et al.  Perception of Words and Pitch Patterns in Song and Speech , 2012, Front. Psychology.

[115]  Christopher L. Asplund,et al.  Functional Specialization and Flexibility in Human Association Cortex. , 2016, Cerebral cortex.

[116]  Robert J Zatorre,et al.  Representations of Invariant Musical Categories Are Decodable by Pattern Analysis of Locally Distributed BOLD Responses in Superior Temporal and Intraparietal Sulci. , 2015, Cerebral cortex.

[117]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[118]  Richard Granger,et al.  Investigation of melodic contour processing in the brain using multivariate pattern-based fMRI , 2011, NeuroImage.

[119]  Jakob Heinzle,et al.  Beyond topographic representation: Decoding visuospatial attention from local activity patterns in the human frontal cortex , 2011, Int. J. Imaging Syst. Technol..

[120]  M. Eckstein,et al.  Feature-Independent Neural Coding of Target Detection during Search of Natural Scenes , 2012, The Journal of Neuroscience.