Decoding neural representational spaces using multivariate pattern analysis.

A major challenge for systems neuroscience is to break the neural code. Computational algorithms for encoding information into neural activity and extracting information from measured activity afford understanding of how percepts, memories, thought, and knowledge are represented in patterns of brain activity. The past decade and a half has seen significant advances in the development of methods for decoding human neural activity, such as multivariate pattern classification, representational similarity analysis, hyperalignment, and stimulus-model-based encoding and decoding. This article reviews these advances and integrates neural decoding methods into a common framework organized around the concept of high-dimensional representational spaces.

[1]  Alexander Borst,et al.  How does Nature Program Neuron Types? , 2008, Front. Neurosci..

[2]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[3]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[4]  J. Haynes Brain Reading: Decoding Mental States From Brain Activity In Humans , 2011 .

[5]  Stefan Pollmann,et al.  PyMVPA: a Python Toolbox for Multivariate Pattern Analysis of fMRI Data , 2009, Neuroinformatics.

[6]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[7]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[8]  Alice J. O'Toole,et al.  Theoretical, Statistical, and Practical Perspectives on Pattern-based Classification Approaches to the Analysis of Functional Neuroimaging Data , 2007, Journal of Cognitive Neuroscience.

[9]  Teuvo Kohonen,et al.  Self-Organizing Maps , 2010 .

[10]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[11]  Hervé Abdi,et al.  How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the Bootstrap and 3-way multidimensional scaling (DISTATIS) , 2009, NeuroImage.

[12]  E. Hillman Coupling mechanism and significance of the BOLD signal: a status report. , 2014, Annual review of neuroscience.

[13]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[14]  R. Passingham,et al.  Reading Hidden Intentions in the Human Brain , 2007, Current Biology.

[15]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[16]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[17]  Johan Wagemans,et al.  Distributed subordinate specificity for bodies, faces, and buildings in human ventral visual cortex , 2010, NeuroImage.

[18]  Karl J. Friston,et al.  Predictive coding under the free-energy principle , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[19]  Simon B Eickhoff,et al.  Meta-analysis in human neuroimaging: computational modeling of large-scale databases. , 2014, Annual review of neuroscience.

[20]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[21]  Tom Michael Mitchell,et al.  From the SelectedWorks of Marcel Adam Just 2008 Using fMRI brain activation to identify cognitive states associated with perception of tools and dwellings , 2016 .

[22]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[23]  T. Aflalo,et al.  Possible Origins of the Complex Topographic Organization of Motor Cortex: Reduction of a Multidimensional Space onto a Two-Dimensional Array , 2006, The Journal of Neuroscience.

[24]  A. Caramazza,et al.  Category-Specific Organization in the Human Brain Does Not Require Visual Experience , 2009, Neuron.

[25]  Johan D. Carlin,et al.  A Head View-Invariant Representation of Gaze Direction in Anterior Superior Temporal Sulcus , 2011, Current Biology.

[26]  M. Brass,et al.  Unconscious determinants of free decisions in the human brain , 2008, Nature Neuroscience.

[27]  Christoph E Schreiner,et al.  Order and disorder in auditory cortical maps , 1995, Current Opinion in Neurobiology.

[28]  Y Kamitani,et al.  Neural Decoding of Visual Imagery During Sleep , 2013, Science.

[29]  T. Carlson,et al.  High temporal resolution decoding of object position and category. , 2011, Journal of vision.

[30]  N. Kanwisher Functional specificity in the human brain: A window into the functional architecture of the mind , 2010, Proceedings of the National Academy of Sciences.

[31]  T. Carlson,et al.  Patterns of Activity in the Categorical Representations of Objects , 2003, Journal of Cognitive Neuroscience.

[32]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[33]  Wei Huang,et al.  Translational control in synaptic plasticity and cognitive dysfunction. , 2014, Annual review of neuroscience.

[34]  Yaroslav O. Halchenko,et al.  Pattern classification precedes region-average hemodynamic response in early visual cortex , 2013, NeuroImage.

[35]  Paul E. Downing,et al.  A comparison of volume-based and surface-based multi-voxel pattern analysis , 2011, NeuroImage.

[36]  P. Schönemann,et al.  A generalized solution of the orthogonal procrustes problem , 1966 .

[37]  Masa-aki Sato,et al.  Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns , 2008, NeuroImage.

[38]  G. Rees,et al.  Predicting the orientation of invisible stimuli from activity in human primary visual cortex , 2005, Nature Neuroscience.

[39]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[40]  Rajeev D. S. Raizada,et al.  What Makes Different People's Representations Alike: Neural Similarity Space Solves the Problem of Across-subject fMRI Decoding , 2012, Journal of Cognitive Neuroscience.

[41]  Richard Durbin,et al.  A dimension reduction framework for understanding cortical maps , 1990, Nature.

[42]  J. S. Guntupalli,et al.  The Representation of Biological Classes in the Human Brain , 2012, The Journal of Neuroscience.

[43]  Hyoung F. Kim,et al.  Basal ganglia circuits for reward value-guided behavior. , 2014, Annual review of neuroscience.

[44]  T. Kohonen Self-Organized Formation of Correct Feature Maps , 1982 .

[45]  David A. van Dyk,et al.  The Role of Statistics in the Discovery of a Higgs Boson , 2014 .

[46]  Hervé Abdi,et al.  STATIS and DISTATIS: optimum multitable principal component analysis and three way metric multidimensional scaling , 2012 .

[47]  N. Kriegeskorte,et al.  Author ' s personal copy Representational geometry : integrating cognition , computation , and the brain , 2013 .

[48]  William Simonson,et al.  Alzheimer's disease update. , 2015, Geriatric nursing.

[49]  Z. Yue,et al.  Autophagy and its normal and pathogenic states in the brain. , 2014, Annual review of neuroscience.

[50]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[51]  Doris Y. Tsao,et al.  Functional Compartmentalization and Viewpoint Generalization Within the Macaque Face-Processing System , 2010, Science.

[52]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[53]  James V. Haxby,et al.  Multivariate pattern analysis of fMRI: The early beginnings , 2012, NeuroImage.

[54]  Steven Laureys,et al.  Measuring consciousness in severely damaged brains. , 2014, Annual review of neuroscience.

[55]  Hong Qian,et al.  Statistics and Related Topics in Single-Molecule Biophysics. , 2014, Annual review of statistics and its application.

[56]  Brian N. Pasley,et al.  Reconstructing Speech from Human Auditory Cortex , 2012, PLoS biology.

[57]  R. Dolmetsch,et al.  Generating human neurons in vitro and using them to understand neuropsychiatric disease. , 2014, Annual review of neuroscience.

[58]  E. Marder,et al.  Neuromodulation of circuits with variable parameters: single neurons and small circuits reveal principles of state-dependent and robust neuromodulation. , 2014, Annual review of neuroscience.

[59]  F. Tong,et al.  Decoding reveals the contents of visual working memory in early visual areas , 2009, Nature.

[60]  M. Ida Gobbini,et al.  Three Virtues of Similarity-based Multivariate Pattern Analysis : An example from the human object vision pathway , 2014 .

[61]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[62]  Anatol C. Kreitzer,et al.  Reassessing models of basal ganglia function and dysfunction. , 2014, Annual review of neuroscience.

[63]  Xiao-Bing Gao,et al.  Function and dysfunction of hypocretin/orexin: an energetics point of view. , 2014, Annual review of neuroscience.

[64]  T. Aflalo,et al.  Mapping Behavioral Repertoire onto the Cortex , 2007, Neuron.

[65]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[66]  B. Mesquita,et al.  Adjustment to Chronic Diseases and Terminal Illness Health Psychology : Psychological Adjustment to Chronic Disease , 2006 .

[67]  Alice J. O'Toole,et al.  Partially Distributed Representations of Objects and Faces in Ventral Temporal Cortex , 2005, Journal of Cognitive Neuroscience.

[68]  Stephen José Hanson,et al.  Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a “face” area? , 2004, NeuroImage.

[69]  Kalanit Grill-Spector,et al.  The improbable simplicity of the fusiform face area , 2012, Trends in Cognitive Sciences.

[70]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[71]  Michael S. Pratte,et al.  Decoding patterns of human brain activity. , 2012, Annual review of psychology.

[72]  Karolina M. Lempert,et al.  Emotion and decision making: multiple modulatory neural circuits. , 2014, Annual review of neuroscience.

[73]  Riitta Salmelin,et al.  Tracking neural coding of perceptual and semantic features of concrete nouns , 2012, NeuroImage.

[74]  Noël Staeren,et al.  Sound Categories Are Represented as Distributed Patterns in the Human Auditory Cortex , 2009, Current Biology.

[75]  Tom M. Mitchell,et al.  Commonality of neural representations of words and pictures , 2011, NeuroImage.

[76]  Bryan R. Conroy,et al.  Function-based Intersubject Alignment of Human Cortical Anatomy , 2009, Cerebral cortex.

[77]  Peter J. Ramadge,et al.  Inter-subject alignment of human cortical anatomy using functional connectivity , 2013, NeuroImage.

[78]  Lloyd T. Elliott,et al.  Cortical surface-based searchlight decoding , 2011, NeuroImage.

[79]  David D. Cox,et al.  Untangling invariant object recognition , 2007, Trends in Cognitive Sciences.

[80]  Hervé Abdi,et al.  Multiple Subject Barycentric Discriminant Analysis (MUSUBADA): How to Assign Scans to Categories without Using Spatial Normalization , 2012, Comput. Math. Methods Medicine.

[81]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[82]  Johan Wagemans,et al.  Multiple scales of organization for object selectivity in ventral visual cortex , 2010, NeuroImage.

[83]  Keiji Tanaka,et al.  Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. , 2007, Journal of neurophysiology.

[84]  Michael A. Casey,et al.  Population Codes Representing Musical Timbre for High-Level fMRI Categorization of Music Genres , 2011, MLINI.

[85]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[86]  Bryan R. Conroy,et al.  A Common, High-Dimensional Model of the Representational Space in Human Ventral Temporal Cortex , 2011, Neuron.

[87]  S. Edelman,et al.  Toward direct visualization of the internal shape representation space by fMRI , 1998, Psychobiology.

[88]  D. Madigan,et al.  A Systematic Statistical Approach to Evaluating Evidence from Observational Studies , 2014 .

[89]  T. Aflalo,et al.  Organization of the macaque extrastriate visual cortex re-examined using the principle of spatial continuity of function. , 2011, Journal of neurophysiology.

[90]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.