Encoding and decoding in fMRI

Over the past decade fMRI researchers have developed increasingly sensitive techniques for analyzing the information represented in BOLD activity. The most popular of these techniques is linear classification, a simple technique for decoding information about experimental stimuli or tasks from patterns of activity across an array of voxels. A more recent development is the voxel-based encoding model, which describes the information about the stimulus or task that is represented in the activity of single voxels. Encoding and decoding are complementary operations: encoding uses stimuli to predict activity while decoding uses activity to predict information about the stimuli. However, in practice these two operations are often confused, and their respective strengths and weaknesses have not been made clear. Here we use the concept of a linearizing feature space to clarify the relationship between encoding and decoding. We show that encoding and decoding operations can both be used to investigate some of the most common questions about how information is represented in the brain. However, focusing on encoding models offers two important advantages over decoding. First, an encoding model can in principle provide a complete functional description of a region of interest, while a decoding model can provide only a partial description. Second, while it is straightforward to derive an optimal decoding model from an encoding model it is much more difficult to derive an encoding model from a decoding model. We propose a systematic modeling approach that begins by estimating an encoding model for every voxel in a scan and ends by using the estimated encoding models to perform decoding.

[1]  R. Passingham,et al.  Reading Hidden Intentions in the Human Brain , 2007, Current Biology.

[2]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[3]  Nikolaus Kriegeskorte,et al.  Analyzing for information, not activation, to exploit high-resolution fMRI , 2007, NeuroImage.

[4]  Dinggang Shen,et al.  Classifying spatial patterns of brain activity with machine learning methods: Application to lie detection , 2005, NeuroImage.

[5]  Chuan Yi Tang,et al.  A 2.|E|-Bit Distributed Algorithm for the Directed Euler Trail Problem , 1993, Inf. Process. Lett..

[6]  Sadato Norihiro,et al.  Visual image reconstruction from human brain activity , 2009 .

[7]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[8]  N. Kanwisher,et al.  Domain specificity in visual cortex. , 2006, Cerebral cortex.

[9]  Russell A. Epstein,et al.  Decoding the Representation of Multiple Simultaneous Objects in Human Occipitotemporal Cortex , 2009, Current Biology.

[10]  A. P. Georgopoulos,et al.  Neuronal population coding of movement direction. , 1986, Science.

[11]  John-Dylan Haynes,et al.  Decoding visual consciousness from human brain signals , 2009, Trends in Cognitive Sciences.

[12]  E. Seidemann,et al.  Optimal decoding of correlated neural population responses in the primate visual cortex , 2006, Nature Neuroscience.

[13]  R. DeCharms Applications of real-time fMRI , 2008, Nature Reviews Neuroscience.

[14]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[15]  N. Kanwisher,et al.  Only some spatial patterns of fMRI response are read out in task performance , 2007, Nature Neuroscience.

[16]  D. Ruppert The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[17]  Benjamin J. Tamber-Rosenau,et al.  Decoding cognitive control in human parietal cortex , 2009, Proceedings of the National Academy of Sciences.

[18]  Lars Kai Hansen,et al.  Massive Weight Sharing: A Cure For Extremely Ill-Posed Problems , 1994 .

[19]  I. Ohzawa,et al.  Receptive Field Properties of Neurons in the Early Visual Cortex Revealed by Local Spectral Reverse Correlation , 2006, The Journal of Neuroscience.

[20]  F. Tong,et al.  Decoding reveals the contents of visual working memory in early visual areas , 2009, Nature.

[21]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[22]  B. Willmore,et al.  Neural Representation of Natural Images in Visual Area V2 , 2010, The Journal of Neuroscience.

[23]  A. Aertsen,et al.  The Spectro-Temporal Receptive Field , 1981, Biological Cybernetics.

[24]  Brian A. Wandell,et al.  Population receptive field estimates in human visual cortex , 2008, NeuroImage.

[25]  Li Fei-Fei,et al.  Neural mechanisms of rapid natural scene categorization in human visual cortex , 2009, Nature.

[26]  Wei Ji Ma,et al.  Bayesian inference with probabilistic population codes , 2006, Nature Neuroscience.

[27]  G. Rees,et al.  Predicting the Stream of Consciousness from Activity in Human Visual Cortex , 2005, Current Biology.

[28]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[29]  B. Duval Commission internationale de l’éclairage (CIE) , 2001, Optique Photonique.

[30]  Kâmil Uğurbil,et al.  Mental maze solving: directional fMRI tuning and population coding in the superior parietal lobule , 2005, Experimental Brain Research.

[31]  J. O'Doherty,et al.  Decoding the neural substrates of reward-related decision making with functional MRI , 2007, Proceedings of the National Academy of Sciences.

[32]  Dirk B. Walther,et al.  Natural Scene Categories Revealed in Distributed Patterns of Activity in the Human Brain , 2009, The Journal of Neuroscience.

[33]  E. Rolls,et al.  Prediction of subjective affective state from brain activations. , 2009, Journal of neurophysiology.

[34]  K. Sen,et al.  Spectral-temporal Receptive Fields of Nonlinear Auditory Neurons Obtained Using Natural Sounds , 2022 .

[35]  A. Ishai,et al.  Recollection- and Familiarity-Based Decisions Reflect Memory Strength , 2008, Frontiers in systems neuroscience.

[36]  Rainer Goebel,et al.  Information-based functional brain mapping. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[37]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[38]  Alice J. O'Toole,et al.  Theoretical, Statistical, and Practical Perspectives on Pattern-based Classification Approaches to the Analysis of Functional Neuroimaging Data , 2007, Journal of Cognitive Neuroscience.

[39]  Nikolaus Kriegeskorte,et al.  Comparison of multivariate classifiers and response normalizations for pattern-information fMRI , 2010, NeuroImage.

[40]  J. Gallant,et al.  Natural Stimulus Statistics Alter the Receptive Field Structure of V1 Neurons , 2004, The Journal of Neuroscience.

[41]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.

[42]  Peter Dayan,et al.  Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems , 2001 .

[43]  J. Kippenhan,et al.  Evaluation of a neural-network classifier for PET scans of normal and Alzheimer's disease subjects. , 1992, Journal of nuclear medicine : official publication, Society of Nuclear Medicine.

[44]  Tom M. Mitchell,et al.  Machine learning classifiers and fMRI: A tutorial overview , 2009, NeuroImage.

[45]  William Bialek,et al.  Spikes: Exploring the Neural Code , 1996 .

[46]  S. Shamma,et al.  Analysis of dynamic spectra in ferret primary auditory cortex. I. Characteristics of single-unit responses to moving ripple spectra. , 1996, Journal of neurophysiology.

[47]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[48]  Tom M. Mitchell,et al.  Learning to Decode Cognitive States from Brain Images , 2004, Machine Learning.

[49]  G. Rees,et al.  Predicting the orientation of invisible stimuli from activity in human primary visual cortex , 2005, Nature Neuroscience.

[50]  Kenneth A. Norman,et al.  Recollection, Familiarity, and Cortical Reinstatement: A Multivoxel Pattern Analysis , 2009, Neuron.

[51]  Rainer Goebel,et al.  Combining multivariate voxel selection and support vector machines for mapping and classification of fMRI spatial patterns , 2008, NeuroImage.

[52]  S. Rossitti Introduction to Functional Magnetic Resonance Imaging, Principles and Techniques , 2002 .

[53]  S. David,et al.  Influence of context and behavior on stimulus reconstruction from neural activity in primary auditory cortex. , 2009, Journal of neurophysiology.

[54]  N. Kriegeskorte,et al.  Revealing representational content with pattern-information fMRI--an introductory guide. , 2009, Social cognitive and affective neuroscience.

[55]  J. Duncan,et al.  Top-Down Activation of Shape-Specific Population Codes in Visual Cortex during Mental Imagery , 2009, The Journal of Neuroscience.

[56]  Jean-Baptiste Poline,et al.  Inferring behavior from functional brain images , 1998, Nature Neuroscience.

[57]  Raymond J. Dolan,et al.  fMRI Activity Patterns in Human LOC Carry Information about Object Exemplars within Category , 2008, Journal of Cognitive Neuroscience.

[58]  D. Heeger,et al.  Decoding and Reconstructing Color from Responses in Human Visual Cortex , 2009, The Journal of Neuroscience.

[59]  Giancarlo Valente,et al.  Multivariate analysis of fMRI time series: classification and regression of brain responses using machine learning. , 2008, Magnetic resonance imaging.

[60]  R. Raizada,et al.  Quantifying the adequacy of neural representations for a cross-language phonetic discrimination task: prediction of individual differences. , 2010, Cerebral cortex.

[61]  Rainer Goebel,et al.  "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech , 2008, Science.

[62]  F. Tong,et al.  Decoding Seen and Attended Motion Directions from Activity in the Human Visual Cortex , 2006, Current Biology.

[63]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[64]  P. H. Schiller,et al.  Spatial frequency and orientation tuning dynamics in area V1 , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[65]  Nicole C. Rust,et al.  Do We Know What the Early Visual System Does? , 2005, The Journal of Neuroscience.

[66]  Karl J. Friston,et al.  Analysis of fMRI Time-Series Revisited , 1995, NeuroImage.

[67]  D. Ringach,et al.  Dynamics of Spatial Frequency Tuning in Macaque V1 , 2002, The Journal of Neuroscience.

[68]  N. Kanwisher,et al.  Mental Imagery of Faces and Places Activates Corresponding Stimulus-Specific Brain Regions , 2000, Journal of Cognitive Neuroscience.

[69]  D. J. Felleman,et al.  Distributed hierarchical processing in the primate cerebral cortex. , 1991, Cerebral cortex.

[70]  J. Haynes Brain Reading: Decoding Mental States From Brain Activity In Humans , 2011 .

[71]  Justin L. Gardner,et al.  Executed and Observed Movements Have Different Distributed Representations in Human aIPS , 2008, The Journal of Neuroscience.

[72]  Mitsuo Kawato,et al.  Sparse linear regression for reconstructing muscle activity from human cortical fMRI , 2008, NeuroImage.

[73]  E. DeYoe,et al.  A physiological correlate of the 'spotlight' of visual attention , 1999, Nature Neuroscience.

[74]  L. K. Hansen,et al.  Multivariate strategies in functional magnetic resonance imaging , 2007, Brain and Language.

[75]  Yoshimichi Yoshimichi,et al.  For Further Development of Flow Visualization: Preface , 1999 .

[76]  John-Dylan Haynes,et al.  Odor quality coding and categorization in human posterior piriform cortex , 2009, Nature Neuroscience.

[77]  T. Carlson,et al.  Patterns of Activity in the Categorical Representations of Objects , 2003, Journal of Cognitive Neuroscience.

[78]  Dario L. Ringach,et al.  Dynamics of orientation tuning in macaque primary visual cortex , 1997, Nature.

[79]  Karl J. Friston,et al.  Bayesian decoding of brain images , 2008, NeuroImage.

[80]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[81]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[82]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[83]  Kendrick N Kay,et al.  I can see what you see , 2009, Nature Neuroscience.

[84]  Stephen José Hanson,et al.  Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a “face” area? , 2004, NeuroImage.

[85]  N. Kanwisher,et al.  The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception , 1997, The Journal of Neuroscience.

[86]  Christian K. Machens,et al.  Linearity of Cortical Receptive Fields Measured with Natural Sounds , 2004, The Journal of Neuroscience.

[87]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[88]  J. Gallant,et al.  Predicting neuronal responses during natural vision , 2005, Network.

[89]  M. Schönwiesner,et al.  Spectro-temporal modulation transfer function of single voxels in the human auditory cortex measured with high-resolution fMRI , 2009, Proceedings of the National Academy of Sciences.

[90]  David D. Cox,et al.  Untangling invariant object recognition , 2007, Trends in Cognitive Sciences.

[91]  M. Brass,et al.  Unconscious determinants of free decisions in the human brain , 2008, Nature Neuroscience.

[92]  Thomas Serre,et al.  Reading the mind's eye: Decoding category information during mental imagery , 2010, NeuroImage.

[93]  Michael S Beauchamp,et al.  Distributed Representation of Single Touches in Somatosensory and Visual Cortex , 2009, NeuroImage.

[94]  Jonathan D. Cohen,et al.  Reproducibility Distinguishes Conscious from Nonconscious Neural Representations , 2010, Science.

[95]  Duane Q Nykamp,et al.  Full identification of a linear-nonlinear system via cross-correlation analysis. , 2002, Journal of vision.

[96]  Dimitri Van De Ville,et al.  Decoding of Emotional Information in Voice-Sensitive Cortices , 2009, Current Biology.

[97]  Steen Moeller,et al.  Ultra-high field parallel imaging of the superior parietal lobule during mental maze solving , 2008, Experimental Brain Research.

[98]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[99]  B L McNaughton,et al.  Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. , 1998, Journal of neurophysiology.

[100]  Karl J. Friston Modalities, Modes, and Models in Functional Neuroimaging , 2009, Science.

[101]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.