Encoding functional brain interactions from computational visual features

Encoding and decoding in functional magnetic resonance imaging (fMRI) are receiving increasing interest recently. In this paper, we propose an fMRI encoding model to predict the human brain's responses to free viewing of video clips. The novelty of our study is that we represent the stimuli using a variety of representative visual features, which can describe the global color distribution, local shape and spatial information contained in videos and they have been proven to be effective by computer vision studies. Our experimental results demonstrate that brain network responses during free viewing of videos can be robustly and accurately predicted by using visual features across subjects. Our study suggests the feasibility to exploring cognitive neuroscience studies by computational image/video analysis and provides a novel concept of using the brain encoding as a test-bed for evaluating visual feature extraction.

[1]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[2]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[3]  Dimitri Van De Ville,et al.  Decoding brain states from fMRI connectivity graphs , 2011, NeuroImage.

[4]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[5]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[6]  Li Fei-Fei,et al.  Neural mechanisms of rapid natural scene categorization in human visual cortex , 2009, Nature.

[7]  D. Shen,et al.  DICCCOL: dense individualized and common connectivity-based cortical landmarks. , 2013, Cerebral cortex.

[8]  Koen E. A. van de Sande,et al.  Evaluating Color Descriptors for Object and Scene Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Vaidehi S. Natu,et al.  Category-Specific Cortical Activity Precedes Retrieval During Memory Search , 2005, Science.

[10]  J. Haynes Brain Reading: Decoding Mental States From Brain Activity In Humans , 2011 .

[11]  Stephen J. Heinen,et al.  The spatial extent of attention for saccades: Attentional facilitation compared to inhibition of return in humans and monkeys , 2010 .

[12]  Arnold W. M. Smeulders,et al.  Color Based Object Recognition , 1997, ICIAP.

[13]  Sean M. Polyn,et al.  Beyond mind-reading: multi-voxel pattern analysis of fMRI data , 2006, Trends in Cognitive Sciences.

[14]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[15]  G. Rees,et al.  Neuroimaging: Decoding mental states from brain activity in humans , 2006, Nature Reviews Neuroscience.

[16]  M. Greicius,et al.  Decoding subject-driven cognitive states with whole-brain connectivity patterns. , 2012, Cerebral cortex.

[17]  G. Rees,et al.  Fine-scale activity patterns in high-level visual areas encode the category of invisible objects. , 2008, Journal of vision.

[18]  N. Logothetis,et al.  Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain. , 2008, Cerebral cortex.