Self-supervised Natural Image Reconstruction and Large-scale Semantic Classification from Brain Activity

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs (image, fMRI) that span the huge space of natural images is prohibitive. We present a novel self-supervised approach for fMRI-to-image reconstruction and classification that goes well beyond the scarce paired data. By imposing cycle consistency, we train our image reconstruction deep neural network on many “unpaired” data: a plethora of natural images without fMRI recordings (from many novel categories), and fMRI recordings without images. Combining high-level perceptual objectives with self-supervision on unpaired data results in a leap improvement over top existing methods, achieving: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing); (ii) Large-scale semantic classification (1000 diverse classes) of categories that are never-before-seen during network training. Such large-scale (1000-way) semantic classification capabilities from fMRI recordings have never been demonstrated before. Finally, we provide evidence for the biological plausibility of our learned model. 1

[1]  A. Caramazza,et al.  Tripartite Organization of the Ventral Stream by Animacy and Object Size , 2013, The Journal of Neuroscience.

[2]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[3]  Matthew H. Davis,et al.  Detecting Awareness in the Vegetative State , 2006, Science.

[4]  E. Switkes,et al.  Deoxyglucose analysis of retinotopic organization in primate striate cortex. , 1982, Science.

[5]  Yizhen Zhang,et al.  Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex , 2017, bioRxiv.

[6]  Luca Ambrogioni,et al.  Generative adversarial networks for reconstructing natural images from brain activity , 2017, NeuroImage.

[7]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[8]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[9]  Guohua Shen,et al.  Deep image reconstruction from human brain activity , 2017, bioRxiv.

[10]  J W Belliveau,et al.  Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. , 1995, Science.

[11]  Fang Sun,et al.  Willful modulation of brain activity in disorders of consciousness. , 2010, The New England journal of medicine.

[12]  Y. Kamitani,et al.  End-to-End Deep Image Reconstruction From Human Brain Activity , 2019, Frontiers Comput. Neurosci..

[13]  Michal Irani,et al.  From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI , 2019, NeurIPS.

[14]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[15]  Miki Haseyama,et al.  Estimating Viewed Image Categories from fMRI Activity via Multi-view Bayesian Generative Model , 2019, 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE).

[16]  Eugenio Culurciello,et al.  Deep Predictive Coding Network for Object Recognition , 2018, ICML.

[17]  Jonathan Winawer,et al.  Computational neuroimaging and population receptive fields , 2015, Trends in Cognitive Sciences.

[18]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[19]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[20]  Michal Irani,et al.  Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks , 2019, Nature Communications.

[21]  N. Kanwisher,et al.  Mental Imagery of Faces and Places Activates Corresponding Stimulus-Specific Brain Regions , 2000, Journal of Cognitive Neuroscience.

[22]  V. Mountcastle.  Modality and topographic properties of single neurons of cat's somatic sensory cortex. , 1957, Journal of neurophysiology.

[23]  Jesse Graham,et al.  Ideological differences in the expanse of the moral circle , 2019, Nature Communications.

[24]  Ghislain St-Yves,et al.  The feature-weighted receptive field: an interpretable encoding model for complex feature spaces , 2017, NeuroImage.

[25]  L. Reddy,et al.  Reconstructing Natural Scenes from fMRI Patterns using BigBiGAN , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[26]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[27]  B. Dow,et al.  Orientation and color columns in monkey visual cortex. , 2002, Cerebral cortex.

[28]  Tomoyasu Horikawa,et al.  Generic decoding of seen and imagined objects using hierarchical visual features , 2015, Nature Communications.

[29]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[30]  Junxing Shi,et al.  Deep Residual Network Predicts Cortical Representation and Organization of Visual Features for Rapid Categorization , 2018, Scientific Reports.

[31]  Li Tong,et al.  BigGAN-based Bayesian Reconstruction of Natural Images from Human Brain Activity , 2020, Neuroscience.

[32]  Yunfeng Lin,et al.  DCNN-GAN: Reconstructing Realistic Image from fMRI , 2019, 2019 16th International Conference on Machine Vision Applications (MVA).

[33]  Chi Zhang,et al.  Constraint-Free Natural Image Reconstruction From fMRI Signals Based on Convolutional Neural Network , 2018, Front. Hum. Neurosci..

[34]  T. Ohshima,et al.  Stimulated emission from nitrogen-vacancy centres in diamond , 2016, Nature Communications.

[35]  Y Kamitani,et al.  Neural Decoding of Visual Imagery During Sleep , 2013, Science.

[36]  Amiram Grinvald,et al.  Iso-orientation domains in cat visual cortex are arranged in pinwheel-like patterns , 1991, Nature.

[37]  R. Goebel,et al.  Brain–computer interfaces for communication with nonresponsive patients , 2012, Annals of neurology.

[38]  Michael Eickenberg,et al.  Seeing it all: Convolutional network layers map the function of the human visual system , 2017, NeuroImage.

[39]  Radoslaw Martin Cichy,et al.  Imagery and perception share cortical representations of content and location. , 2012, Cerebral Cortex.

[40]  G. Glover.  Overview of functional magnetic resonance imaging. , 2011, Neurosurgery clinics of North America.

[41]  Mark Jenkinson,et al.  The minimal preprocessing pipelines for the Human Connectome Project , 2013, NeuroImage.

[42]  Mark A. Anastasio,et al.  Decoding visual information from high-density diffuse optical tomography neuroimaging data , 2018, NeuroImage.

[43]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[44]  Jack L. Gallant,et al.  A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes , 2015, NeuroImage.

[45]  Anish A. Sarma,et al.  Stable long-term BCI-enabled communication in ALS and locked-in syndrome using LFP signals. , 2018, Journal of neurophysiology.

[46]  A. Janowsky,et al.  Role of serotonergic input in the regulation of the beta-adrenergic receptor-coupled adenylate cyclase system. , 1982, Science.

[47]  Chris I. Baker,et al.  Deconstructing multivariate decoding for the study of brain function , 2017, NeuroImage.

[48]  Yizhen Zhang,et al.  Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision , 2016, Cerebral cortex.

[49]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  Vaidehi S. Natu,et al.  Development differentially sculpts receptive fields across early and high-level human visual cortex , 2018, Nature Communications.

[51]  Elia Formisano,et al.  Methods for computing the maximum performance of computational models of fMRI responses , 2018, bioRxiv.

[52]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Wei Chen,et al.  Transferring and generalizing deep-learning-based neural encoding models across subjects , 2017, NeuroImage.

[54]  Xinbo Gao,et al.  Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning , 2021, NeuroImage.

[55]  Daniel Y. Ts’o,et al.  Whither the hypercolumn? , 2009, The Journal of physiology.

[56]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[57]  Li Tong,et al.  Category Decoding of Visual Stimuli From Human Brain Activity Using a Bidirectional Recurrent Neural Network to Simulate Bidirectional Information Flows in Human Visual Cortices , 2019, Front. Neurosci..

[58]  Tomoyasu Horikawa,et al.  Hierarchical Neural Representation of Dreamed Objects Revealed by Brain Decoding with Deep Neural Network Features , 2016, Front. Comput. Neurosci..

[59]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.

[60]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[61]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[62]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[63]  S. Kosslyn,et al.  Mental Imagery: Functional Mechanisms and Clinical Applications , 2015, Trends in Cognitive Sciences.

[64]  F. Tong,et al.  Decoding Seen and Attended Motion Directions from Activity in the Human Visual Cortex , 2006, Current Biology.

[65]  Ghislain St-Yves,et al.  Generative Adversarial Networks Conditioned on Brain Activity Reconstruct Seen Images , 2018, bioRxiv.