Multi-head attention-based masked sequence model for mapping functional brain networks

The investigation of functional brain networks (FBNs) using task-based functional magnetic resonance imaging (tfMRI) has gained significant attention in the field of neuroimaging. Despite the availability of several methods for constructing FBNs, including traditional methods like GLM and deep learning methods such as spatiotemporal self-attention mechanism (STAAE), these methods have design and training limitations. Specifically, they do not consider the intrinsic characteristics of fMRI data, such as the possibility that the same signal value at different time points could represent different brain states and meanings. Furthermore, they overlook prior knowledge, such as task designs, during training. This study aims to overcome these limitations and develop a more efficient model by drawing inspiration from techniques in the field of natural language processing (NLP). The proposed model, called the Multi-head Attention-based Masked Sequence Model (MAMSM), uses a multi-headed attention mechanism and mask training approach to learn different states corresponding to the same voxel values. Additionally, it combines cosine similarity and task design curves to construct a novel loss function. The MAMSM was applied to seven task state datasets from the Human Connectome Project (HCP) tfMRI dataset. Experimental results showed that the features acquired by the MAMSM model exhibit a Pearson correlation coefficient with the task design curves above 0.95 on average. Moreover, the model can extract more meaningful networks beyond the known task-related brain networks. The experimental results demonstrated that MAMSM has great potential in advancing the understanding of functional brain networks.

[1]  Yu Zhao,et al.  Characterizing functional brain networks via Spatio-Temporal Attention 4D Convolutional Neural Networks (STA-4DCNNs) , 2022, Neural Networks.

[2]  J. Lv,et al.  Modeling spatio-temporal patterns of holistic functional brain networks via multi-head guided attention graph neural networks (Multi-Head GAGNNs) , 2022, Medical Image Anal..

[3]  Limin Wang,et al.  VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training , 2022, NeurIPS.

[4]  Trevor Darrell,et al.  A ConvNet for the 2020s , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Han Hu,et al.  SimMIM: a Simple Framework for Masked Image Modeling , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Tao Kong,et al.  iBOT: Image BERT Pre-Training with Online Tokenizer , 2021, ArXiv.

[7]  Ross B. Girshick,et al.  Masked Autoencoders Are Scalable Vision Learners , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Chung-Cheng Chiu,et al.  w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training , 2021, 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU).

[9]  Bharat B. Biswal,et al.  A review of resting-state fMRI and its use to examine psychiatric disorders , 2021, Psychoradiology.

[10]  Douwe Kiela,et al.  Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little , 2021, EMNLP.

[11]  K. Kendrick,et al.  Fundamental functional differences between gyri and sulci: implications for brain function, cognition, and behavior , 2021, Psychoradiology.

[12]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[13]  Xiang Li,et al.  Spatiotemporal Attention Autoencoder (STAAE) for ADHD Classification , 2020, MICCAI.

[14]  Xiang Li,et al.  Discovering Functional Brain Networks with 3D Residual Autoencoder (ResAE) , 2020, MICCAI.

[15]  Tianming Liu,et al.  Deep Variational Autoencoder for Mapping Functional Brain Networks , 2020, IEEE Transactions on Cognitive and Developmental Systems.

[16]  Jinglei Lv,et al.  Experimental Comparisons of Sparse Dictionary Learning and Independent Component Analysis for Brain Network Inference From fMRI Data , 2019, IEEE Transactions on Biomedical Engineering.

[17]  Xiang Li,et al.  Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN) , 2018, MICCAI.

[18]  Yu Zhao,et al.  Modeling Task fMRI Data Via Deep Convolutional Autoencoder , 2018, IEEE Transactions on Medical Imaging.

[19]  Dewen Hu,et al.  Making group inferences using sparse representation of resting‐state functional mRI data with application to sleep deprivation , 2017, Human brain mapping.

[20]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[21]  Marcel van Gerven,et al.  Modeling the Dynamics of Human Brain Activity with Recurrent Neural Networks , 2016, Front. Comput. Neurosci..

[22]  Jinglei Lv,et al.  Characterizing and differentiating task-based and resting state fMRI signals via two-stage sparse representations , 2016, Brain Imaging and Behavior.

[23]  Sang Won Seo,et al.  Sparse SPM: Group Sparse-dictionary learning in SPM framework for resting-state functional connectivity MRI analysis , 2016, NeuroImage.

[24]  Richard F. Betzel,et al.  Modular Brain Networks. , 2016, Annual review of psychology.

[25]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[26]  Jinglei Lv,et al.  Characterizing and differentiating task-based and resting state FMRI signals via two-stage dictionary learning , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[27]  Junwei Han,et al.  Signal sampling for efficient sparse representation of resting state FMRI data , 2015, 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI).

[28]  Jieping Ye,et al.  Holistic Atlases of Functional Networks and Interactions Reveal Reciprocal Organizational Architecture of Cortical Function , 2015, IEEE Transactions on Biomedical Engineering.

[29]  Heng Huang,et al.  Sparse representation of whole-brain fMRI signals for identification of functional networks , 2015, Medical Image Anal..

[30]  Aapo Hyvärinen,et al.  Group-PCA for very large fMRI datasets , 2014, NeuroImage.

[31]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[32]  Karl J. Friston,et al.  Structural and Functional Brain Networks: From Connections to Cognition , 2013, Science.

[33]  Abraham Z. Snyder,et al.  Function in the human connectome: Task-fMRI and individual differences in behavior , 2013, NeuroImage.

[34]  Essa Yacoub,et al.  The WU-Minn Human Connectome Project: An overview , 2013, NeuroImage.

[35]  V. Calhoun,et al.  Multisubject Independent Component Analysis of fMRI: A Decade of Intrinsic Networks, Default Mode, and Neurodiagnostic Discovery , 2012, IEEE Reviews in Biomedical Engineering.

[36]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[37]  Jonathan D. Power,et al.  The Development of Human Functional Brain Networks , 2010, Neuron.

[38]  Janaina Mourão Miranda,et al.  The impact of temporal compression and space selection on SVM analysis of single-subject and multi-subject fMRI data , 2006, NeuroImage.

[39]  Jürgen Schmidhuber,et al.  Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition , 2005, ICANN.

[40]  Stephen C. Strother,et al.  Support vector machines for temporal classification of block design fMRI data , 2005, NeuroImage.

[41]  Stephen M. Smith,et al.  Investigations into resting-state connectivity using independent component analysis , 2005, Philosophical Transactions of the Royal Society B: Biological Sciences.

[42]  Stephen M. Smith,et al.  General multilevel linear modeling for group analysis in FMRI , 2003, NeuroImage.

[43]  Olivier Faugeras,et al.  Dynamical components analysis of fMRI data through kernel PCA , 2003, NeuroImage.

[44]  M. McKeown.  Detection of Consistently Task-Related Activations in fMRI Data with Hybrid Independent Component Analysis , 2000, NeuroImage.

[45]  S. Hochreiter,et al.  Long Short-Term Memory , 1997, Neural Computation.

[46]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[47]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[48]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.