Decoding the Brain: Neural Representation and the Limits of Multivariate Pattern Analysis in Cognitive Neuroscience

Since its introduction, multivariate pattern analysis (MVPA), or “neural decoding”, has transformed the field of cognitive neuroscience. Underlying its influence is a crucial inference, which we call the Decoder’s Dictum: if information can be decoded from patterns of neural activity, then this provides strong evidence about what information those patterns represent. Although the Dictum is a widely held and well-motivated principle in decoding research, it has received scant philosophical attention. We critically evaluate the Dictum, arguing that it is false: decodability is a poor guide for revealing the content of neural representations. However, we also suggest how the Dictum can be improved on, in order to better justify inferences about neural representation using MVPA.

[1]  Russell A. Poldrack,et al.  What do differences between multi-voxel and univariate analysis mean? How subject-, voxel-, and trial-level variance impact fMRI analysis , 2014, NeuroImage.

[2]  H. P. Op de Beeck,et al.  Dissociations and Associations between Shape and Category Representations in the Two Visual Pathways , 2015, The Journal of Neuroscience.

[3]  Krish D. Singh,et al.  Which “neural activity” do you mean? fMRI, MEG, oscillations and neurotransmitters , 2012, NeuroImage.

[4]  Thomas A. Carlson,et al.  Emerging Object Representations in the Visual System Predict Reaction Times for Categorization , 2015, PLoS Comput. Biol..

[5]  Nikolaus Kriegeskorte,et al.  Comparison of multivariate classifiers and response normalizations for pattern-information fMRI , 2010, NeuroImage.

[6]  Peter Carruthers,et al.  Opening Up Vision: The Case Against Encapsulation , 2016 .

[7]  F. Tong,et al.  Decoding reveals the contents of visual working memory in early visual areas , 2009, Nature.

[8]  Fred I. Dretske,et al.  Précis of Knowledge and the Flow of Information , 1983, Behavioral and Brain Sciences.

[9]  R. Vogels,et al.  Inferotemporal neurons represent low-dimensional configurations of parameterized shapes , 2001, Nature Neuroscience.

[10]  Nikolaus Kriegeskorte,et al.  Analyzing for information, not activation, to exploit high-resolution fMRI , 2007, NeuroImage.

[11]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[12]  N. Kanwisher,et al.  Only some spatial patterns of fMRI response are read out in task performance , 2007, Nature Neuroscience.

[13]  F ATTNEAVE,et al.  Dimensions of similarity. , 1950, The American journal of psychology.

[14]  N. Logothetis What we can do and what we cannot do with fMRI , 2008, Nature.

[15]  S. Dehaene,et al.  Characterizing the dynamics of mental representations: the temporal generalization method , 2014, Trends in Cognitive Sciences.

[16]  J. Macke,et al.  Neural population coding: combining insights from microscopic and mass signals , 2015, Trends in Cognitive Sciences.

[17]  Nikolaus Kriegeskorte,et al.  Reaction Time for Object Categorization Is Predicted by Representational Distance , 2014, Journal of Cognitive Neuroscience.

[18]  Colin Klein,et al.  Philosophical Issues in Neuroimaging , 2010 .

[19]  P. Hanson Information, language, and cognition , 1993 .

[20]  R. Poldrack Can cognitive processes be inferred from neuroimaging data? , 2006, Trends in Cognitive Sciences.

[21]  P. Sajda,et al.  Temporal characterization of the neural correlates of perceptual decision making in the human brain. , 2006, Cerebral cortex.

[22]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[23]  Colin W. G. Clifford,et al.  Discrimination of the local orientation structure of spiral Glass patterns early in human visual cortex , 2009, NeuroImage.

[24]  Nicholas G Hatsopoulos,et al.  The science of neural interface systems. , 2009, Annual review of neuroscience.

[25]  Jeremy Freeman,et al.  Orientation Decoding Depends on Maps, Not Columns , 2011, The Journal of Neuroscience.

[26]  R. Dawes Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[27]  Omar H. Butt,et al.  The Retinotopic Organization of Striate Cortex Is Well Predicted by Surface Topology , 2012, Current Biology.

[28]  William Bechtel,et al.  Representations and Cognitive Explanations: Assessing the Dynamicist's Challenge in Cognitive Science , 1998, Cogn. Sci..

[29]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[30]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[31]  J. S. Guntupalli,et al.  Decoding neural representational spaces using multivariate pattern analysis. , 2014, Annual review of neuroscience.

[32]  David D. Cox,et al.  Untangling invariant object recognition , 2007, Trends in Cognitive Sciences.

[33]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[34]  W. T. Maddox,et al.  A response time theory of separability and integrality in speeded classification , 1994 .

[35]  Nicholas J. Priebe,et al.  Mechanisms of Neuronal Computation in Mammalian Visual Cortex , 2012, Neuron.

[36]  Janneke F. M. Jehee,et al.  Attention Improves Encoding of Task-Relevant Features in the Human Visual Cortex , 2011, The Journal of Neuroscience.

[37]  Chris Eliasmith,et al.  Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems , 2004, IEEE Transactions on Neural Networks.

[38]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[39]  Doris Y. Tsao,et al.  Single-Unit Recordings in the Macaque Face Patch System Reveal Limitations of fMRI MVPA , 2015, The Journal of Neuroscience.

[40]  D. Heeger,et al.  BOLD and spiking activity , 2008, Nature Neuroscience.

[41]  N. Kriegeskorte,et al.  Author ' s personal copy Representational geometry : integrating cognition , computation , and the brain , 2013 .

[42]  Keiji Tanaka,et al.  Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. , 2007, Journal of neurophysiology.

[43]  N. Kriegeskorte,et al.  Revealing representational content with pattern-information fMRI--an introductory guide. , 2009, Social cognitive and affective neuroscience.

[44]  Andreas Bartels,et al.  The Coding of Color, Motion, and Their Conjunction in the Human Visual Cortex , 2009, Current Biology.

[45]  Janneke F. M. Jehee,et al.  Less Is More: Expectation Sharpens Representations in the Primary Visual Cortex , 2012, Neuron.

[46]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[47]  James V. Haxby,et al.  Multivariate pattern analysis of fMRI: The early beginnings , 2012, NeuroImage.

[48]  R. Shepard Attention and the metric structure of the stimulus space. , 1964 .

[49]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[50]  Yevgeniy B. Sirotin,et al.  Anticipatory haemodynamic signals in sensory cortex not predicted by local neuronal activity. , 2009, Nature.

[51]  David D. Cox,et al.  Do we understand high-level vision? , 2014, Current Opinion in Neurobiology.

[52]  N. Logothetis,et al.  Neurophysiological investigation of the basis of the fMRI signal , 2001, Nature.

[53]  R. Poldrack,et al.  Measuring neural representations with fMRI: practices and pitfalls , 2013, Annals of the New York Academy of Sciences.

[54]  Nikolaus Kriegeskorte,et al.  Perceptual similarity of visual patterns predicts dynamic neural activation patterns measured with MEG , 2016, NeuroImage.

[55]  Takeshi Norimatsu,et al.  Encoding and Decoding , 2016 .

[56]  G. Rees,et al.  Predicting the orientation of invisible stimuli from activity in human primary visual cortex , 2005, Nature Neuroscience.

[57]  R. Goebel,et al.  Human Object-Similarity Judgments Reflect and Transcend the Primate-IT Object Representation , 2013, Front. Psychol..

[58]  John Duncan,et al.  Multi-voxel coding of stimuli, rules, and responses in human frontoparietal cortex , 2011, NeuroImage.

[59]  Stefania Bracci,et al.  Dissociations and associations between shape and category representations in the two visual pathways. , 2015, Journal of vision.

[60]  R. Poldrack,et al.  Quantifying the internal structure of categories using a neural typicality measure. , 2014, Cerebral cortex.

[61]  F. Tong,et al.  Decoding Seen and Attended Motion Directions from Activity in the Human Visual Cortex , 2006, Current Biology.

[62]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[63]  Thomas A. Carlson,et al.  Neural Decoding and “Inner” Psychophysics: A Distance-to-Bound Approach for Linking Mind, Brain, and Behavior , 2016, Front. Neurosci..

[64]  G. Piccinini,et al.  Information without Truth , 2010 .

[65]  David Kirsh,et al.  When is Information Explicitly Represented , 1990 .

[66]  S. Edelman,et al.  Toward direct visualization of the internal shape representation space by fMRI , 1998, Psychobiology.

[67]  A. Zador,et al.  Neural representation and the cortical code. , 2000, Annual review of neuroscience.

[68]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[69]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[70]  Michael S. Pratte,et al.  Decoding patterns of human brain activity. , 2012, Annual review of psychology.

[71]  N. Kanwisher,et al.  Feedback of pVisual Object Information to Foveal Retinotopic Cortex , 2008, Nature Neuroscience.

[72]  Yaroslav O. Halchenko,et al.  The Animacy Continuum in the Human Ventral Vision Pathway , 2015, Journal of Cognitive Neuroscience.

[73]  Dae-Shik Kim,et al.  Global and local fMRI signals driven by neurons defined optogenetically by type and wiring , 2010, Nature.

[74]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .