Peeling the Onion of Brain Representations.

The brain's function is to enable adaptive behavior in the world. To this end, the brain processes information about the world. The concept of representation links the information processed by the brain back to the world and enables us to understand what the brain does at a functional level. The appeal of making the connection between brain activity and what it represents has been irresistible to neuroscience, despite the fact that representational interpretations pose several challenges: We must define which aspects of brain activity matter, how the code works, and how it supports computations that contribute to adaptive behavior. It has been suggested that we might drop representational language altogether and seek to understand the brain, more simply, as a dynamical system. In this review, we argue that the concept of representation provides a useful link between dynamics and computational function and ask which aspects of brain activity should be analyzed to achieve a representational understanding. We peel the onion of brain representations in search of the layers (the aspects of brain activity) that matter to computation. The article provides an introduction to the motivation and mathematics of representational models, a critical discussion of their assumptions and limitations, and a preview of future directions in this area.

[1]  William T. Newsome,et al.  Cortical microstimulation influences perceptual judgements of motion direction , 1990, Nature.

[2]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[3]  A. Dale,et al.  From retinotopy to recognition: fMRI in human visual cortex , 1998, Trends in Cognitive Sciences.

[4]  S. Edelman,et al.  Toward direct visualization of the internal shape representation space by fMRI , 1998, Psychobiology.

[5]  S Edelman,et al.  Representation is representation of similarities , 1996, Behavioral and Brain Sciences.

[6]  William Bechtel,et al.  Representations and Cognitive Explanations: Assessing the Dynamicist's Challenge in Cognitive Science , 1998, Cogn. Sci..

[7]  T. Gelder,et al.  The dynamical hypothesis in cognitive science , 1998, Behavioral and Brain Sciences.

[8]  Peter Dayan,et al.  The Effect of Correlated Variability on the Accuracy of a Population Code , 1999, Neural Computation.

[9]  A. Zador,et al.  Neural representation and the cortical code. , 2000, Annual review of neuroscience.

[10]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[11]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[12]  R. Goebel,et al.  Mirror-Symmetric Tonotopic Maps in Human Primary Auditory Cortex , 2003, Neuron.

[13]  T. Carlson,et al.  Patterns of Activity in the Categorical Representations of Objects , 2003, Journal of Cognitive Neuroscience.

[14]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[15]  Bruno A Olshausen,et al.  Sparse coding of sensory inputs , 2004, Current Opinion in Neurobiology.

[16]  D. Chklovskii,et al.  Maps in the brain: what can we learn from them? , 2004, Annual review of neuroscience.

[17]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[18]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[19]  K. Grill-Spector,et al.  Repetition and the brain: neural models of stimulus-specific effects , 2006, Trends in Cognitive Sciences.

[20]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[21]  R. Kiani,et al.  Microstimulation of inferotemporal cortex influences face categorization , 2006, Nature.

[22]  D. Chklovskii,et al.  Wiring optimization can relate neuronal structure and function. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[24]  A. Pouget,et al.  Neural correlations, population coding and computation , 2006, Nature Reviews Neuroscience.

[25]  Nikolaus Kriegeskorte,et al.  Analyzing for information, not activation, to exploit high-resolution fMRI , 2007, NeuroImage.

[26]  R. Goebel,et al.  Individual faces elicit distinct response patterns in human anterior temporal cortex , 2007, Proceedings of the National Academy of Sciences.

[27]  David D. Cox,et al.  Opinion TRENDS in Cognitive Sciences Vol.11 No.8 Untangling invariant object recognition , 2022 .

[28]  Liam Paninski,et al.  Statistical models for neural encoding, decoding, and optimal stimulus design. , 2007, Progress in brain research.

[29]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[30]  John P. Cunningham,et al.  Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity , 2008, NIPS.

[31]  Brian A. Wandell,et al.  Population receptive field estimates in human visual cortex , 2008, NeuroImage.

[32]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[33]  Nikolaus Kriegeskorte,et al.  Representational Similarity Analysis – Connecting the Branches of Systems Neuroscience , 2008, Frontiers in systems neuroscience.

[34]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[35]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[36]  N. Kriegeskorte,et al.  Revealing representational content with pattern-information fMRI--an introductory guide. , 2009, Social cognitive and affective neuroscience.

[37]  Nikolaus Kriegeskorte,et al.  Comparison of multivariate classifiers and response normalizations for pattern-information fMRI , 2010, NeuroImage.

[38]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[39]  Nikolaus Kriegeskorte,et al.  Pattern-information analysis: From stimulus decoding to computational-model testing , 2011, NeuroImage.

[40]  Karl J. Friston,et al.  Comparing the similarity and spatial structure of neural representations: A pattern-component model , 2011, NeuroImage.

[41]  J. Haynes Brain Reading: Decoding Mental States From Brain Activity In Humans , 2011 .

[42]  H. Sompolinsky,et al.  Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis. , 2012, Annual review of neuroscience.

[43]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[44]  Michael S. Pratte,et al.  Decoding patterns of human brain activity. , 2012, Annual review of psychology.

[45]  K. Grill-Spector,et al.  Electrical Stimulation of Human Fusiform Face-Selective Regions Distorts Face Perception , 2012, The Journal of Neuroscience.

[46]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[47]  D. Poeppel The maps problem and the mapping problem: Two challenges for a cognitive neuroscience of speech and language , 2012, Cognitive neuropsychology.

[48]  Matthew T. Kaufman,et al.  Neural population dynamics during reaching , 2012, Nature.

[49]  E. Marder,et al.  From the connectome to brain function , 2013, Nature Methods.

[50]  M. Sahani,et al.  Cortical control of arm movements: a dynamical systems perspective. , 2013, Annual review of neuroscience.

[51]  N. Kriegeskorte,et al.  Representational geometry: integrating cognition, computation, and the brain , 2013, Trends in Cognitive Sciences.

[52]  Radoslaw Martin Cichy,et al.  Resolving human object recognition in space and time , 2014, Nature Neuroscience.

[53]  J. S. Guntupalli,et al.  Decoding neural representational spaces using multivariate pattern analysis. , 2014, Annual review of neuroscience.

[54]  A. Pouget,et al.  Information-limiting correlations , 2014, Nature Neuroscience.

[55]  Li Su,et al.  A Toolbox for Representational Similarity Analysis , 2014, PLoS Comput. Biol..

[56]  Byron M. Yu,et al.  Dimensionality reduction for large-scale neural recordings , 2014, Nature Neuroscience.

[57]  S. Dehaene,et al.  Characterizing the dynamics of mental representations: the temporal generalization method , 2014, Trends in Cognitive Sciences.

[58]  John-Dylan Haynes,et al.  Searchlight-based multi-voxel pattern analysis of fMRI by cross-validated MANOVA , 2014, NeuroImage.

[59]  J. Diedrichsen,et al.  Hand use predicts the structure of representations in sensorimotor cortex , 2015, Nature Neuroscience.

[60]  Thomas Naselaris,et al.  Resolving Ambiguities of MVPA Using Explicit Models of Representation , 2015, Trends in Cognitive Sciences.

[61]  J. Haynes A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives , 2015, Neuron.

[62]  Stuart P. Wilson,et al.  What, if anything, are topological maps for? , 2015, Developmental neurobiology.

[63]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[64]  Josh H. McDermott,et al.  Distinct Cortical Pathways for Music and Speech Revealed by Hypothesis-Free Voxel Decomposition , 2015, Neuron.

[65]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[66]  Surya Ganguli,et al.  On simplicity and complexity in the brave new world of large-scale neuroscience , 2015, Current Opinion in Neurobiology.

[67]  Ha Hong,et al.  Explicit information for category-orthogonal object properties increases along the ventral stream , 2016, Nature Neuroscience.

[68]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[69]  Jörn Diedrichsen,et al.  Reliability of dissimilarity measures for multi-voxel pattern analysis , 2016, NeuroImage.

[70]  J. Diedrichsen,et al.  On the distribution of cross-validated Mahalanobis distances , 2016, 1607.01371.

[71]  Naoshige Uchida,et al.  Demixed principal component analysis of neural population data , 2016, eLife.

[72]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[73]  Jörn Diedrichsen,et al.  Inferring brain-computational mechanisms with models of activity measurements , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[74]  Jonathan W. Pillow,et al.  A Bayesian method for reducing bias in neural representational similarity analysis , 2016, bioRxiv.

[75]  Marcel A. J. van Gerven,et al.  A primer on encoding models in sensory neuroscience , 2017 .

[76]  Jörn Diedrichsen,et al.  Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis , 2017, bioRxiv.

[77]  Andres Hoyos Idrobo,et al.  Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines , 2016, NeuroImage.

[78]  Jörn Diedrichsen,et al.  Pattern component modeling: A flexible approach for understanding the representational structure of brain activity patterns , 2017, NeuroImage.

[79]  L .Paninski,et al.  Neural data science: accelerating the experiment-analysis-theory cycle in large-scale neuroscience , 2017, Current Opinion in Neurobiology.

[80]  Nikolaus Kriegeskorte,et al.  Cognitive computational neuroscience , 2018, Nature Neuroscience.

[81]  Chris I. Baker,et al.  Deconstructing multivariate decoding for the study of brain function , 2017, NeuroImage.

[82]  J. Diedrichsen Representational models and the feature fallacy , 2018 .