Integrating Space, Time, and Orientation in Spiking Neural Networks: A Case Study on Multimodal Brain Data Modeling

Recent progress in a noninvasive brain data sampling technology has facilitated simultaneous sampling of multiple modalities of brain data, such as functional magnetic resonance imaging, electroencephalography, diffusion tensor imaging, and so on. In spite of the potential benefits from integrating predictive modeling of multiple modality brain data, this area of research remains mostly unexplored due to a lack of methodological advancements. The difficulty in fusing multiple modalities of brain data within a single model lies in the heterogeneous temporal and spatial characteristics of the data sources. Recent advances in spiking neural network systems, however, provide the flexibility to incorporate multidimensional information within the model. This paper proposes a novel, unsupervised learning algorithm for fusing temporal, spatial, and orientation information in a spiking neural network architecture that could potentially be used to understand and perform predictive modeling using multimodal data. The proposed algorithm is evaluated both qualitatively and quantitatively using synthetically generated data to characterize its behavior and its ability to utilize spatial, temporal, and orientation information within the model. This leads to improved pattern recognition capabilities and performance along with robust interpretability of the brain data. Furthermore, a case study is presented, which aims to build a computational model that discriminates between people with schizophrenia who respond or do not respond to monotherapy with the antipsychotic clozapine.

[1]  Jie Yang,et al.  Mapping Temporal Variables Into the NeuCube for Improved Pattern Recognition, Predictive Modeling, and Understanding of Stream Data , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[2]  Jürgen Schmidhuber,et al.  LSTM can Solve Hard Long Time Lag Problems , 1996, NIPS.

[3]  J. Bower,et al.  Cerebellum Implicated in Sensory Acquisition and Discrimination Rather Than Motor Control , 1996, Science.

[4]  Peter Boesiger,et al.  Combining fMRI and DTI: A framework for exploring the limits of fMRI-guided DTI fiber tracking and for verifying DTI-based fiber tractography results , 2008, NeuroImage.

[5]  Vince D. Calhoun,et al.  Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI , 2010, NeuroImage.

[6]  Richard J. Caselli,et al.  Linking functional and structural brain images with multivariate network analyses: A novel application of the partial least square method , 2009, NeuroImage.

[7]  Maryam Gholami Doborjeh,et al.  Mapping, Learning, Visualization, Classification, and Understanding of fMRI Data in the NeuCube Evolving Spatiotemporal Data Machine of Spiking Neural Networks , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Timothy Edward John Behrens,et al.  Characterization and propagation of uncertainty in diffusion‐weighted MR imaging , 2003, Magnetic resonance in medicine.

[9]  Nikola K. Kasabov,et al.  Spike-time encoding as a data compression technique for pattern recognition of temporal data , 2017, Inf. Sci..

[10]  P. Strick,et al.  Anatomical evidence for cerebellar and basal ganglia involvement in higher cognitive function. , 1994, Science.

[11]  J. Reilly,et al.  A pilot study to determine whether machine learning methodologies using pre-treatment electroencephalography can predict the symptomatic response to clozapine therapy , 2010, Clinical Neurophysiology.

[12]  P. Strick,et al.  Activation of a cerebellar output nucleus during cognitive processing. , 1994, Science.

[13]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[14]  N. Akshoomoff,et al.  A model system for the study of attention and the cerebellum: infantile autism. , 1995, Electroencephalography and clinical neurophysiology. Supplement.

[15]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[16]  Maarten Mennes,et al.  Evaluation of ICA-AROMA and alternative strategies for motion artifact removal in resting state fMRI , 2015, NeuroImage.

[17]  Jose M. Sanchez-Bornot,et al.  Model driven EEG/fMRI fusion of brain oscillations , 2009, Human brain mapping.

[18]  Susan P. Worner,et al.  NeuCube(ST) for spatio-temporal data predictive modelling with a case study on ecological data , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[19]  Adeel Razi,et al.  A DCM for resting state fMRI , 2014, NeuroImage.

[20]  Vince D. Calhoun,et al.  Effective connectivity analysis of fMRI and MEG data collected under identical paradigms , 2011, Comput. Biol. Medicine.

[21]  Adeel Razi,et al.  Dynamic causal modelling revisited , 2017, NeuroImage.

[22]  Anna Vilanova,et al.  CUDA-Accelerated Geodesic Ray-Tracing for Fiber Tracking , 2011, Int. J. Biomed. Imaging.

[23]  Vince D. Calhoun,et al.  Human Neuroscience , 2022 .

[24]  Vince D. Calhoun,et al.  A review of multivariate methods for multimodal fusion of brain imaging data , 2012, Journal of Neuroscience Methods.

[25]  Wulfram Gerstner,et al.  A neuronal learning rule for sub-millisecond temporal coding , 1996, Nature.

[26]  Vince D. Calhoun,et al.  Examining associations between FMRI and EEG data using canonical correlation analysis , 2008, 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[27]  Maryam Gholami Doborjeh,et al.  Evolving spatio-temporal data machines based on the NeuCube neuromorphic framework: Design methodology and selected applications , 2016, Neural Networks.

[28]  Karl J. Friston,et al.  Dysconnection in Schizophrenia: From Abnormal Synaptic Plasticity to Failures of Self-monitoring , 2009, Schizophrenia bulletin.

[29]  Nikola Kasabov,et al.  New Algorithms for Encoding, Learning and Classification of fMRI Data in a Spiking Neural Network Architecture: A Case on Modeling and Understanding of Dynamic Cognitive Processes , 2017, IEEE Transactions on Cognitive and Developmental Systems.

[30]  Olaf Sporns,et al.  Complex network measures of brain connectivity: Uses and interpretations , 2010, NeuroImage.

[31]  J. Price,et al.  Machine learning approaches for integrating clinical and imaging features in late‐life depression classification and response prediction , 2015, International journal of geriatric psychiatry.

[32]  D. Louis Collins,et al.  Unbiased average age-appropriate atlases for pediatric studies , 2011, NeuroImage.

[33]  Nikola Kasabov,et al.  Framework for Knowledge Driven Optimisation Based Data Encoding for Brain Data Modelling Using Spiking Neural Network Architecture , 2015 .

[34]  I. Yang.  Impact of fMRI-guided advanced DTI fiber tracking techniques on their clinical applications in patients with brain tumors , 2011 .

[35]  B. Schrauwen,et al.  BSA, a fast and accurate spike train encoding scheme , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[36]  Tijmen Tieleman,et al.  Training restricted Boltzmann machines using approximations to the likelihood gradient , 2008, ICML '08.

[37]  Yu-Chuan Li,et al.  Artificial neural network prediction of clozapine response with combined pharmacogenetic and clinical data , 2008, Comput. Methods Programs Biomed..

[38]  P. Falkai,et al.  The role of the cerebellum in schizophrenia: from cognition to molecular pathways , 2011, Clinics.

[39]  Wulfram Gerstner,et al.  Spike-timing dependent plasticity , 2010, Scholarpedia.

[40]  Nikola K. Kasabov,et al.  EEG Classification with BSA Spike Encoding Algorithm and Evolving Probabilistic Spiking Neural Network , 2011, ICONIP.

[41]  Mark W. Woolrich,et al.  FSL , 2012, NeuroImage.

[42]  Vince D. Calhoun,et al.  Canonical Correlation Analysis for Feature-Based Fusion of Biomedical Imaging Modalities and Its Application to Detection of Associative Networks in Schizophrenia , 2008, IEEE Journal of Selected Topics in Signal Processing.

[43]  David Poeppel,et al.  How can EEG/MEG and fMRI/PET data be combined? , 2002, Human brain mapping.

[44]  Peter Mariën,et al.  Cerebellar cognitive affective syndrome associated with topiramate , 2008, Clinical Neurology and Neurosurgery.

[45]  Maximilian Reiser,et al.  White matter microstructure underlying default mode network connectivity in the human brain , 2010, NeuroImage.

[46]  Karl J. Friston,et al.  Dynamic causal modeling , 2010, Scholarpedia.

[47]  Ahmad Khodayari-Rostamabad,et al.  A machine learning approach using EEG data to predict response to SSRI treatment for major depressive disorder , 2013, Clinical Neurophysiology.

[48]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[49]  Muhaini Othman,et al.  Evolving spiking neural networks for personalised modelling, classification and prediction of spatio-temporal patterns with a case study on stroke , 2014, Neurocomputing.

[50]  Vince D. Calhoun,et al.  A review of group ICA for fMRI data and ICA for joint inference of imaging, genetic, and ERP data , 2009, NeuroImage.

[51]  Nikola K. Kasabov,et al.  NeuCube: A spiking neural network architecture for mapping, learning and understanding of spatio-temporal brain data , 2014, Neural Networks.

[52]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[53]  M. Greicius,et al.  Resting-state functional connectivity reflects structural connectivity in the default mode network. , 2009, Cerebral cortex.

[54]  Satrajit S. Ghosh,et al.  Predicting treatment response in social anxiety disorder from functional magnetic resonance imaging. , 2012, JAMA psychiatry.

[55]  Alberto Llera,et al.  ICA-AROMA: A robust ICA-based strategy for removing motion artifacts from fMRI data , 2015, NeuroImage.

[56]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[57]  Nikola Kasabov,et al.  Dynamic evolving spiking neural networks for on-line spatio- and spectro-temporal pattern recognition. , 2013, Neural networks : the official journal of the International Neural Network Society.