Brain Decoding of Viewed Image Categories via Semi-Supervised Multi-View Bayesian Generative Model

Brain decoding has shown that viewed image categories can be estimated from evoked functional magnetic resonance imaging (fMRI) activity. Recent studies attempted to estimate viewed image categories that were not used for training previously. Nevertheless, the estimation performance is limited since it is difficult to collect a large amount of fMRI data for training. This paper presents a method to accurately estimate viewed image categories not used for training via a semi-supervised multi-view Bayesian generative model. Our model focuses on the relationship between fMRI activity and multiple modalities, i.e., visual features extracted from viewed images and semantic features obtained from viewed image categories. Furthermore, in order to accurately estimate image categories not used for training, our semi-supervised framework incorporates visual and semantic features obtained from additional image categories in addition to image categories of training data. The estimation performance of the proposed model outperforms existing state-of-the-art models in the brain decoding field and achieves more than 95% identification accuracy. The results also have shown that the incorporation of additional image category information is remarkably effective when the number of training samples is small. Our semi-supervised framework is significant for the brain decoding field where brain activity patterns are insufficient but visual stimuli are sufficient.

[1]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Yukiyasu Kamitani,et al.  Modular Encoding and Decoding Models Derived from Bayesian Canonical Correlation Analysis , 2013, Neural Computation.

[4]  Changde Du,et al.  Semi-supervised cross-modal image generation with generative adversarial networks , 2020, Pattern Recognit..

[5]  W. K. Simmons,et al.  Circular analysis in systems neuroscience: the dangers of double dipping , 2009, Nature Neuroscience.

[6]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[7]  Miguel A. L. Nicolelis,et al.  Actions from thoughts , 2001, Nature.

[8]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[9]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[10]  Tomoyasu Horikawa,et al.  Generic decoding of seen and imagined objects using hierarchical visual features , 2015, Nature Communications.

[11]  Masa-aki Sato,et al.  Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns , 2008, NeuroImage.

[12]  Samuel Kaski,et al.  Group Factor Analysis , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[13]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[14]  Shinji Nishimoto,et al.  Decoding naturalistic experiences from human brain activity via distributed representations of words , 2017, NeuroImage.

[15]  J. Daugman.  Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[16]  Shin Ishii,et al.  A Bayesian missing value estimation method for gene expression profile data , 2003, Bioinform..

[17]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[18]  Anastasios Tefas,et al.  Visual representation decoding from human brain activity using machine learning: A baseline study , 2019, Pattern Recognit. Lett..

[19]  Miki Haseyama,et al.  Estimating Viewed Image Categories from fMRI Activity via Multi-view Bayesian Generative Model , 2019, 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE).

[20]  Mitsuo Kawato,et al.  Sparse linear regression for reconstructing muscle activity from human cortical fMRI , 2008, NeuroImage.

[21]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[22]  Wiebke Wagner,et al.  Steven Bird, Ewan Klein and Edward Loper: Natural Language Processing with Python, Analyzing Text with the Natural Language Toolkit , 2010, Lang. Resour. Evaluation.

[23]  Kei Majima,et al.  Brain hierarchy score: Which deep neural networks are hierarchically brain-like? , 2020 .

[24]  Changde Du,et al.  Reconstructing Perceived Images From Human Brain Activities With Bayesian Deep Multiview Learning , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Luca Ambrogioni,et al.  Generative adversarial networks for reconstructing natural images from brain activity , 2017, NeuroImage.

[26]  Miki Haseyama,et al.  Estimation of Viewed Image Categories via CCA Using Human Brain Activity , 2018, 2018 IEEE 7th Global Conference on Consumer Electronics (GCCE).

[27]  Guohua Shen,et al.  Deep image reconstruction from human brain activity , 2017, bioRxiv.

[28]  Yizhen Zhang,et al.  Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex , 2019, NeuroImage.

[29]  Marcel A. J. van Gerven,et al.  Semantic vector space models predict neural responses to complex visual stimuli , 2015, 1510.04738.

[30]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[31]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[32]  Miki Haseyama,et al.  Estimating Viewed Image Categories from Human Brain Activity via Semi-supervised Fuzzy Discriminative Canonical Correlation Analysis , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[33]  Tom Heskes,et al.  Neural Decoding with Hierarchical Generative Models , 2010, Neural Computation.

[34]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[35]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[36]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[38]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[39]  Thomas Serre,et al.  Reading the mind's eye: Decoding category information during mental imagery , 2010, NeuroImage.

[40]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[41]  Michal Irani,et al.  From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI , 2019, NeurIPS.

[42]  Chong Wang,et al.  Variational Bayesian Approach to Canonical Correlation Analysis , 2007, IEEE Transactions on Neural Networks.

[43]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[44]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[45]  Tomoyasu Horikawa,et al.  Characterization of deep neural network features by decodability from human brain activity , 2019, Scientific Data.

[46]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[47]  Tom Heskes,et al.  Linear reconstruction of perceived images from human brain activity , 2013, NeuroImage.