Modeling the Dynamics of Human Brain Activity with Recurrent Neural Networks

Encoding models are used for predicting brain activity in response to sensory stimuli with the objective of elucidating how sensory information is represented in the brain. Encoding models typically comprise a nonlinear transformation of stimuli to features (feature model) and a linear convolution of features to responses (response model). While there has been extensive work on developing better feature models, the work on developing better response models has been rather limited. Here, we investigate the extent to which recurrent neural network models can use their internal memories for nonlinear processing of arbitrary feature sequences to predict feature-evoked response sequences as measured by functional magnetic resonance imaging. We show that the proposed recurrent neural network models can significantly outperform established response models by accurately estimating long-term dependencies that drive hemodynamic responses. The results open a new window into modeling the dynamics of brain activity in response to sensory stimuli.

[1]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[2]  M. W. Pedersen,et al.  Training recurrent networks , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[3]  Marcel van Gerven,et al.  Increasingly complex representations of natural movies across the dorsal stream are shared between subjects , 2017, NeuroImage.

[4]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  G. Glover.  Deconvolution of Impulse Response in Event-Related BOLD fMRI1 , 1999, NeuroImage.

[7]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[8]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[9]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[10]  G. Altmann.  Learning and development in neural networks – the importance of prior experience , 2002, Cognition.

[11]  Tom M. Mitchell,et al.  Selecting Corpus-Semantic Models for Neurolinguistic Decoding , 2012, *SEMEVAL.

[12]  Michael Eickenberg,et al.  Seeing it all: Convolutional network layers map the function of the human visual system , 2017, NeuroImage.

[13]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[14]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[15]  Wojciech Zaremba,et al.  Recurrent Neural Network Regularization , 2014, ArXiv.

[16]  Nikos K Logothetis,et al.  Interpreting the BOLD signal. , 2004, Annual review of physiology.

[17]  B. Wandell,et al.  Compressive spatial summation in human visual cortex. , 2013, Journal of neurophysiology.

[18]  Daniel L. K. Yamins,et al.  Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition , 2014, PLoS Comput. Biol..

[19]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[20]  Karl J. Friston,et al.  Nonlinear event‐related responses in fMRI , 1998, Magnetic resonance in medicine.

[21]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[22]  Karl J. Friston,et al.  Dynamic causal modelling , 2003, NeuroImage.

[23]  Antonio Torralba,et al.  Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition , 2016, ArXiv.

[24]  Erhardt Barth,et al.  Recurrent Dropout without Memory Loss , 2016, COLING.

[25]  Jitendra Malik,et al.  Pixels to Voxels: Modeling Visual Representation in the Human Brain , 2014, ArXiv.

[26]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[27]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[28]  J. Haynes.  A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives , 2015, Neuron.

[29]  Stephen V. David,et al.  Parametric reverse correlation reveals spatial linearity of retinotopic human V1 BOLD response , 2004, NeuroImage.

[30]  Karl J. Friston,et al.  Nonlinear Responses in fMRI: The Balloon Model, Volterra Kernels, and Other Hemodynamics , 2000, NeuroImage.

[31]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[32]  Tom M. Mitchell,et al.  Documents and Dependencies: an Exploration of Vector Space Models for Semantic Composition , 2013, CoNLL.

[33]  D. Norris.  Principles of magnetic resonance assessment of brain function , 2006, Journal of magnetic resonance imaging : JMRI.

[34]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[35]  James J DiCarlo,et al.  Eight open questions in the computational modeling of higher sensory cortex , 2016, Current Opinion in Neurobiology.

[36]  In-Seuck Jeung,et al.  Investigation of the pseudo-shock wave in a two-dimensional supersonic inlet , 2010, J. Vis..

[37]  Marcel van Gerven,et al.  Unsupervised Feature Learning Improves Prediction of Human Brain Activity in Response to Natural Images , 2014, PLoS Comput. Biol..

[38]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[39]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[40]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[41]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[42]  Michael Eickenberg,et al.  Data-driven HRF estimation for encoding and decoding models , 2014, NeuroImage.

[43]  Pradeep Ravikumar,et al.  ENCODING AND DECODING V1 FMRI RESPONSES TO NATURAL IMAGES WITH SPARSE NONPARAMETRIC MODELS. , 2011, The annals of applied statistics.

[44]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[45]  Tom Michael Mitchell,et al.  Predicting Human Brain Activity Associated with the Meanings of Nouns , 2008, Science.

[46]  Ingmar Kanitscheider,et al.  Training recurrent networks to generate hypotheses about how the brain solves hard navigation problems , 2016, NIPS.

[47]  Sennay Ghebreab,et al.  From Image Statistics to Scene Gist: Evoked Neural Activity Reveals Transition from Low-Level Natural Image Structure to Scene Category , 2013, The Journal of Neuroscience.

[48]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[49]  Leslie G. Ungerleider,et al.  Object vision and spatial vision: two cortical pathways , 1983, Trends in Neurosciences.

[50]  Andrew W. Senior,et al.  Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition , 2014, ArXiv.

[51]  Gidon Felsen,et al.  A natural approach to studying vision , 2005, Nature Neuroscience.

[52]  Karl J. Friston,et al.  Statistical parametric maps in functional imaging: A general linear approach , 1994 .

[53]  P A Robinson,et al.  Spatiotemporal hemodynamic response functions derived from physiology. , 2014, Journal of theoretical biology.

[54]  A M Dale,et al.  Optimal experimental design for event‐related fMRI , 1999, Human brain mapping.

[55]  J. Elman.  Learning and development in neural networks: the importance of starting small , 1993, Cognition.

[56]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[57]  John A. Pyles,et al.  Comparing visual representations across human fMRI and computational vision. , 2013, Journal of vision.

[58]  Gary G. R. Green,et al.  Calculation of the Volterra kernels of non-linear dynamic systems using an artificial neural network , 1994, Biological Cybernetics.

[59]  Fei-Fei Li,et al.  Visualizing and Understanding Recurrent Networks , 2015, ArXiv.

[60]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[61]  Marcel A. J. van Gerven,et al.  Semantic vector space models predict neural responses to complex visual stimuli , 2015, 1510.04738.