Multi-View Bayesian Generative Model for Multi-Subject FMRI Data on Brain Decoding of Viewed Image Categories

Brain decoding studies have demonstrated that viewed image categories can be estimated from human functional magnetic resonance imaging (fMRI) activity. However, there are still limitations with the estimation performance because of the characteristics of fMRI data and the employment of only one modality extracted from viewed images. In this paper, we propose a multi-view Bayesian generative model for multi-subject fMRI data to estimate viewed image categories from fMRI activity. The proposed method derives effective representations of fMRI activity by utilizing multi-subject fMRI data. In addition, we associate fMRI activity with multiple modalities, i.e., visual features and semantic features extracted from viewed images. Experimental results show that the proposed method outperforms existing state-of-the-art methods of brain decoding.

[1]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[2]  Tomoyasu Horikawa,et al.  Generic decoding of seen and imagined objects using hierarchical visual features , 2015, Nature Communications.

[3]  Samuel Kaski,et al.  Group Factor Analysis , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[5]  Miki Haseyama,et al.  Estimating Viewed Image Categories from Human Brain Activity via Semi-supervised Fuzzy Discriminative Canonical Correlation Analysis , 2019, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Changde Du,et al.  Brain Encoding and Decoding in fMRI with Bidirectional Deep Generative Models , 2019, Engineering.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[9]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[10]  Hagai Attias,et al.  Inferring Parameters and Structure of Latent Variable Models by Variational Bayes , 1999, UAI.

[11]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[12]  David D. Cox,et al.  Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex , 2003, NeuroImage.

[13]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[14]  Bryan R. Conroy,et al.  A Common, High-Dimensional Model of the Representational Space in Human Ventral Temporal Cortex , 2011, Neuron.

[15]  Daoqiang Zhang,et al.  Deep Hyperalignment , 2017, NIPS.

[16]  Miki Haseyama,et al.  Estimation of Viewed Image Categories via CCA Using Human Brain Activity , 2018, 2018 IEEE 7th Global Conference on Consumer Electronics (GCCE).

[17]  Daoqiang Zhang,et al.  Local Discriminant Hyperalignment for multi-subject fMRI data alignment , 2017, AAAI.

[18]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[19]  Anastasios Tefas,et al.  Visual representation decoding from human brain activity using machine learning: A baseline study , 2019, Pattern Recognit. Lett..

[20]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[21]  Tomoyasu Horikawa,et al.  Characterization of deep neural network features by decodability from human brain activity , 2019, Scientific Data.

[22]  A. Ishai,et al.  Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal Cortex , 2001, Science.

[23]  Shinji Nishimoto,et al.  Decoding naturalistic experiences from human brain activity via distributed representations of words , 2017, NeuroImage.

[24]  Thomas Serre,et al.  Reading the mind's eye: Decoding category information during mental imagery , 2010, NeuroImage.

[25]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[26]  Po-Hsuan Chen,et al.  A Reduced-Dimension fMRI Shared Response Model , 2015, NIPS.

[27]  Masa-aki Sato,et al.  Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns , 2008, NeuroImage.

[28]  Yukiyasu Kamitani,et al.  Modular Encoding and Decoding Models Derived from Bayesian Canonical Correlation Analysis , 2013, Neural Computation.

[29]  Marcel A. J. van Gerven,et al.  Semantic vector space models predict neural responses to complex visual stimuli , 2015, 1510.04738.

[30]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.