Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI

Functional magnetic resonance imaging (fMRI) data and electroencephalography (EEG) data provide complementary spatio-temporal information about brain function. Methods to couple the relative strengths of these modalities usually involve two stages: first forming a feature set from each dataset based on one criterion followed by exploration of connections among the features using a second criterion. We propose a data fusion method for simultaneously acquired fMRI and EEG data that combines these steps using a single criterion for finding the cross-modality associations and performing source separation. Using multi-set canonical correlation analysis (M-CCA), we obtain a decomposition of the two modalities, into spatial maps for fMRI data and a corresponding temporal evolution for EEG data, based on trial-to-trial covariation across the two modalities. Additionally, the analysis is performed on data from a group of subjects in order to make group inferences about the covariation across modalities. Being multivariate, the proposed method facilitates the study of brain connectivity along with localization of brain function. M-CCA can be easily extended to incorporate different data types and additional modalities. We demonstrate the promise of the proposed method in finding covarying trial-to-trial amplitude modulations (AMs) in an auditory task involving implicit pattern learning. The results show approximately linear decreasing trends in AMs for both modalities and the corresponding spatial activations occur mainly in motor, frontal, temporal, inferior parietal, and orbito-frontal areas that are linked both to sensory function as well as learning and expectation--all of which match activations related to the presented paradigm.

[1]  H. Hotelling Relations Between Two Sets of Variates , 1936 .

[2]  A. Engel,et al.  Trial-by-Trial Coupling of Concurrent Electroencephalogram and Functional Magnetic Resonance Imaging Identifies the Dynamics of Performance Monitoring , 2005, The Journal of Neuroscience.

[3]  A. Murat Tekalp,et al.  Audiovisual Synchronization and Fusion Using Canonical Correlation Analysis , 2007, IEEE Transactions on Multimedia.

[4]  T. Sejnowski,et al.  Removing electroencephalographic artifacts by blind source separation. , 2000, Psychophysiology.

[5]  Vince D Calhoun,et al.  fMRI in an oddball task: effects of target-to-target interval. , 2005, Psychophysiology.

[6]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[7]  R. Quian Quiroga,et al.  Single-trial event-related potentials with wavelet denoising , 2003, Clinical Neurophysiology.

[8]  Richard G. Shiavi,et al.  Determining the Presence of Bias Error Using Statistical Methods , 2009, IEEE Transactions on Information Technology in Biomedicine.

[9]  Friedrich T. Sommer,et al.  Exploratory analysis and data modeling in functional neuroimaging , 2003 .

[10]  Greg G. Brown,et al.  Dysregulation of working memory and default‐mode networks in schizophrenia using independent component analysis, an fBIRN and MCIC study , 2009, Human brain mapping.

[11]  M. Fox,et al.  The global signal and observed anticorrelated resting state brain networks. , 2009, Journal of neurophysiology.

[12]  N. Tzourio-Mazoyer,et al.  Automated Anatomical Labeling of Activations in SPM Using a Macroscopic Anatomical Parcellation of the MNI MRI Single-Subject Brain , 2002, NeuroImage.

[13]  Vince D. Calhoun,et al.  Feature-Based Fusion of Medical Imaging Data , 2009, IEEE Transactions on Information Technology in Biomedicine.

[14]  Vince D. Calhoun,et al.  Group Study of Simulated Driving fMRI Data by Multiset Canonical Correlation Analysis , 2009, NeuroImage.

[15]  M. Scherg,et al.  Localizing P300 Generators in Visual Target and Distractor Processing: A Combined Event-Related Potential and Functional Magnetic Resonance Imaging Study , 2004, The Journal of Neuroscience.

[16]  R. Quiroga,et al.  Unmixing concurrent EEG-fMRI with parallel independent component analysis. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[17]  A. M. Dale,et al.  Spatiotemporal Brain Imaging of Visual-Evoked Activity Using Interleaved EEG and fMRI Recordings , 2001, NeuroImage.

[18]  Tzyy-Ping Jung,et al.  Having your voxels and timing them too , 2003 .

[19]  Vince D. Calhoun,et al.  Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[20]  Todd C. Handy,et al.  Event-related potentials : a methods handbook , 2005 .

[21]  Kenneth Hugdahl,et al.  Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[22]  Vince D. Calhoun,et al.  Neuronal chronometry of target detection: Fusion of hemodynamic and event-related potential data , 2005, NeuroImage.

[23]  Bettina Sorger,et al.  Improved quality of auditory event-related potentials recorded simultaneously with 3-T fMRI: Removal of the ballistocardiogram artefact , 2007, NeuroImage.

[24]  Hans Knutsson,et al.  Exploratory fMRI Analysis by Autocorrelation Maximization , 2002, NeuroImage.

[25]  Mark S. Cohen,et al.  Acquiring simultaneous EEG and functional MRI , 2000, Clinical Neurophysiology.

[26]  U Hegerl,et al.  Event-related potentials: Do they reflect central serotonergic neurotransmission and do they predict clinical response to serotonin agonists? , 2001 .

[27]  V. Calhoun,et al.  Joint independent component analysis for simultaneous EEG-fMRI: principle and simulation. , 2008, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[28]  N. Kiang,et al.  Acoustic noise during functional magnetic resonance imaging. , 2000, The Journal of the Acoustical Society of America.

[29]  Vince D. Calhoun,et al.  Joint Blind Source Separation by Multiset Canonical Correlation Analysis , 2009, IEEE Transactions on Signal Processing.

[30]  B. Biswal,et al.  Functional connectivity of default mode network components: Correlation, anticorrelation, and causality , 2009, Human brain mapping.

[31]  Zhichun Mu,et al.  Feature Fusion Method Based on KCCA for Ear and Profile Face Based Multimodal Recognition , 2007, 2007 IEEE International Conference on Automation and Logistics.

[32]  Pheng-Ann Heng,et al.  Feature fusion method based on canonical correlation analysis and handwritten character recognition , 2004, ICARCV 2004 8th Control, Automation, Robotics and Vision Conference, 2004..

[33]  T. Adali,et al.  A group study of simulated driving fMRI data by multi-set canonical correlation analysis , 2009, NeuroImage.

[34]  Shaogang Gong,et al.  Beyond Facial Expressions: Learning Human Emotion from Body Gestures , 2007, BMVC.

[35]  Fumikazu Miwakeichi,et al.  Concurrent EEG/fMRI analysis by multiway Partial Least Squares , 2004, NeuroImage.

[36]  Vince D. Calhoun,et al.  Canonical Correlation Analysis for Feature-Based Fusion of Biomedical Imaging Modalities and Its Application to Detection of Associative Networks in Schizophrenia , 2008, IEEE Journal of Selected Topics in Signal Processing.

[37]  David Poeppel,et al.  How can EEG/MEG and fMRI/PET data be combined? , 2002, Human brain mapping.

[38]  T. Adali,et al.  A feature-based approach to combine functional MRI, structural MRI and EEG brain imaging data , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[39]  Maurizio Corbetta,et al.  The human brain is intrinsically organized into dynamic, anticorrelated functional networks. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[40]  J. Kettenring,et al.  Canonical Analysis of Several Sets of Variables , 2022 .

[41]  A K Liu,et al.  Spatiotemporal imaging of human brain activity using functional MRI constrained magnetoencephalography data: Monte Carlo simulations. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[42]  Vince D. Calhoun,et al.  CCA for joint blind source separation of multiple datasets with application to group FMRI analysis , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[43]  Lei Cao,et al.  A Novel Feature Fusion Approach Based on Blocking and Its Application in Image Recognition , 2006, ICIC.

[44]  R. Bowtell,et al.  “sparse” temporal sampling in auditory fMRI , 1999, Human brain mapping.