Identifying natural images from human brain activity

A challenging goal in neuroscience is to be able to read out, or decode, mental content from brain activity. Recent functional magnetic resonance imaging (fMRI) studies have decoded orientation, position and object category from activity in visual cortex. However, these studies typically used relatively simple stimuli (for example, gratings) or images drawn from fixed categories (for example, faces, houses), and decoding was based on previous measurements of brain activity evoked by those same stimuli or categories. To overcome these limitations, here we develop a decoding method based on quantitative receptive-field models that characterize the relationship between visual stimuli and fMRI activity in early visual areas. These models describe the tuning of individual voxels for space, orientation and spatial frequency, and are estimated directly from responses evoked by natural images. We show that these receptive-field models make it possible to identify, from a large set of completely novel natural images, which specific image was seen by an observer. Identification is not a mere consequence of the retinotopic organization of visual areas; simpler receptive-field models that describe only spatial tuning yield much poorer identification performance. Our results suggest that it may soon be possible to reconstruct a picture of a person’s visual experience from measurements of brain activity alone.

[1]  J. Daugman.  Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[2]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[3]  A. Cuevas,et al.  A comparative study of several smoothing methods in density estimation , 1994 .

[4]  C. Goutis,et al.  Estimation in linear models using gradient descent with early stopping , 1994 .

[5]  Adrian T. Lee,et al.  fMRI of human visual cortex , 1994, Nature.

[6]  J W Belliveau,et al.  Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. , 1995, Science.

[7]  Tai Sing Lee,et al.  Image Representation Using 2D Gabor Wavelets , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  E. DeYoe,et al.  Mapping striate and extrastriate visual areas in human cerebral cortex. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[9]  D. Heeger,et al.  Linear Systems Analysis of Functional Magnetic Resonance Imaging in Human V1 , 1996, The Journal of Neuroscience.

[10]  Toby Berger,et al.  Reliable On-Line Human Signature Verification Systems , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  A. Dale,et al.  Functional Analysis of V3A and Related Areas in Human Visual Cortex , 1997, The Journal of Neuroscience.

[12]  B L McNaughton,et al.  Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. , 1998, Journal of neurophysiology.

[13]  A. Dale,et al.  Functional analysis of primary visual cortex (V1) in humans. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[14]  G B Stanley,et al.  Reconstruction of Natural Scenes from Ensemble Responses in the Lateral Geniculate Nucleus , 1999, The Journal of Neuroscience.

[15]  Ning Qian,et al.  On the momentum term in gradient descent learning algorithms , 1999, Neural Networks.

[16]  A M Dale,et al.  Optimal experimental design for event‐related fMRI , 1999, Human brain mapping.

[17]  Karl J. Friston,et al.  A direct quantitative relationship between the functional properties of human and macaque V5 , 2000, Nature Neuroscience.

[18]  D. G. Albrecht,et al.  Spikes versus BOLD: what does neuroimaging tell us about neuronal activity? , 2000, Nature Neuroscience.

[19]  N. Kanwisher,et al.  Mental Imagery of Faces and Places Activates Corresponding Stimulus-Specific Brain Regions , 2000, Journal of Cognitive Neuroscience.

[20]  Ravi S. Menon,et al.  BOLD fMRI response of early visual areas to perceived contrast in human amblyopia. , 2000, Journal of neurophysiology.

[21]  A. T. Smith,et al.  Spatiotemporal Frequency and Direction Sensitivities of Human Visual Areas Measured Using fMRI , 2000, NeuroImage.

[22]  C. Furmanski,et al.  An oblique effect in human primary visual cortex , 2000, Nature Neuroscience.

[23]  Nikos K. Logothetis,et al.  Nonmonotonic noise tuning of BOLD fMRI signal to natural images in the visual cortex of the anesthetized monkey , 2001, Current Biology.

[24]  T A Carpenter,et al.  Colored noise and computational inference in neurophysiological (fMRI) time series analysis: Resampling methods in time and wavelet domains , 2001, Human brain mapping.

[25]  A. T. Smith,et al.  Estimating receptive field size from fMRI data in human striate and extrastriate visual cortex. , 2001, Cerebral cortex.

[26]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[27]  David C. Van Essen,et al.  Application of Information Technology: An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex , 2001, J. Am. Medical Informatics Assoc..

[28]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[29]  D. Heeger,et al.  In this issue , 2002, Nature Reviews Drug Discovery.

[30]  R. Savoy.  Functional Magnetic Resonance Imaging (fMRI) , 2002 .

[31]  Geoffrey M. Boynton,et al.  Efficient Design of Event-Related fMRI Experiments Using M-Sequences , 2002, NeuroImage.

[32]  Jacco A. de Zwart,et al.  Method for functional MRI mapping of nonlinear response , 2003, NeuroImage.

[33]  G. Boynton,et al.  Orientation-Specific Adaptation in Human Visual Cortex , 2003, The Journal of Neuroscience.

[34]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[35]  Emilio Salinas,et al.  Vector reconstruction from firing rates , 1994, Journal of Computational Neuroscience.

[36]  Nikos K Logothetis,et al.  Interpreting the BOLD signal. , 2004, Annual review of physiology.

[37]  R. Lotto,et al.  Responses of human visual cortex to uniform surfaces , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[38]  Stephen V. David,et al.  Parametric reverse correlation reveals spatial linearity of retinotopic human V1 BOLD response , 2004, NeuroImage.

[39]  Paul Schrater,et al.  BOLD fMRI and psychophysical measurements of contrast response to broadband images , 2004, Vision Research.

[40]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[41]  Alice J. O'Toole,et al.  Partially Distributed Representations of Objects and Faces in Ventral Temporal Cortex , 2005, Journal of Cognitive Neuroscience.

[42]  G. Rees,et al.  Predicting the Stream of Consciousness from Activity in Human Visual Cortex , 2005, Current Biology.

[43]  Mark M. Schira,et al.  fMRI localizer technique: Efficient acquisition and functional properties of single retinotopic positions in the human visual cortex , 2005, NeuroImage.

[44]  Stephen A Engel,et al.  Adaptation of Oriented and Unoriented Color-Selective Neurons in Human Visual Areas , 2005, Neuron.

[45]  G. Rees,et al.  Predicting the orientation of invisible stimuli from activity in human primary visual cortex , 2005, Nature Neuroscience.

[46]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[47]  Linda Henriksson,et al.  Multifocal fMRI mapping of visual cortical areas , 2005, NeuroImage.

[48]  D. Kersten,et al.  Orientation-tuned FMRI adaptation in human visual cortex. , 2005, Journal of neurophysiology.

[49]  M. Landy,et al.  Orientation-selective adaptation to first- and second-order patterns in human visual cortex. , 2006, Journal of neurophysiology.

[50]  Doris Y. Tsao,et al.  A Cortical Region Consisting Entirely of Face-Selective Cells , 2006, Science.

[51]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[52]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[53]  M. Sereno,et al.  A human parietal face area contains aligned head-centered visual and tactile maps , 2006, Nature Neuroscience.

[54]  Wim Vanduffel,et al.  The Radial Bias: A Different Slant on Visual Orientation Sensitivity in Human and Nonhuman Primates , 2006, Neuron.

[55]  Daniel Kersten,et al.  Spatially specific FMRI repetition effects in human visual cortex. , 2006, Journal of neurophysiology.

[56]  D. Heeger,et al.  Two Retinotopic Visual Areas in Human Lateral Occipital Cortex , 2006, The Journal of Neuroscience.

[57]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.

[58]  B. Wandell,et al.  Visual Field Maps in Human Cortex , 2007, Neuron.

[59]  Kathleen A. Hansen,et al.  Topographic Organization in and near Human Visual Area V4 , 2007, The Journal of Neuroscience.

[60]  Brian A. Wandell,et al.  Population receptive field estimates in human visual cortex , 2008, NeuroImage.

[61]  Kathleen A. Hansen,et al.  Modeling low‐frequency fluctuation and hemodynamic response timecourse in event‐related fMRI , 2008, Human brain mapping.