Visual image reconstruction from human brain activity: A modular decoding approach

Brain activity represents our perceptual experience. But the potential for reading out perceptual contents from human brain activity has not been fully explored. In this study, we demonstrate constraint-free reconstruction of visual images perceived by a subject, from the brain activity pattern. We reconstructed visual images by combining local image bases with multiple scales, whose contrasts were independently decoded from fMRI activity by automatically selecting relevant voxels and exploiting their correlated patterns. Binary-contrast, 10 x 10-patch images (2100 possible states), were accurately reconstructed without any image prior by measuring brain activity only for several hundred random images. The results suggest that our approach provides an effective means to read out complex perceptual states from brain activity while discovering information representation in multi-voxel patterns.

[1]  J. Robson,et al.  Application of fourier analysis to the visibility of gratings , 1968, The Journal of physiology.

[2]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.

[3]  Brian A. Wandell,et al.  Population receptive field estimates in human visual cortex , 2008, NeuroImage.

[4]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[5]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[6]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[7]  S. Kosslyn,et al.  Topographical representations of mental images in primary visual cortex , 1995, Nature.

[8]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[9]  Robert O. Duncan,et al.  Cortical Magnification within Human Primary Visual Cortex Correlates with Acuity Thresholds , 2003, Neuron.

[10]  A. Pouget,et al.  Neural correlations, population coding and computation , 2006, Nature Reviews Neuroscience.

[11]  Tomaso Poggio,et al.  Generalization in vision and motor control , 2004, Nature.

[12]  G B Stanley,et al.  Reconstruction of Natural Scenes from Ensemble Responses in the Lateral Geniculate Nucleus , 1999, The Journal of Neuroscience.

[13]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[14]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[15]  J W Belliveau,et al.  Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. , 1995, Science.

[16]  D. Hubel,et al.  Receptive fields and functional architecture of monkey striate cortex , 1968, The Journal of physiology.

[17]  B. Fischer,et al.  Visual field representations and locations of visual areas V1/2/3 in human visual cortex. , 2003, Journal of vision.

[18]  Masa-aki Sato,et al.  Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns , 2008, NeuroImage.

[19]  E. Seidemann,et al.  Optimal decoding of correlated neural population responses in the primate visual cortex , 2006, Nature Neuroscience.

[20]  F. Tong,et al.  Decoding Seen and Attended Motion Directions from Activity in the Human Visual Cortex , 2006, Current Biology.

[21]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[22]  D. G. Albrecht,et al.  Spatial frequency selectivity of cells in macaque visual cortex , 1982, Vision Research.