A Convolutional Neural Network for Transcoding Simultaneously Acquired EEG-fMRI Data

In this paper, we use convolutional neural networks (CNNs) to model/capture the relationship between simultaneously acquired EEG and fMRI. Specifically we use CNNs to implement "neural transcoding" - i.e. generating one neuroimaging modality from another - from EEG to fMRI and vice versa. The novelty of our approach lies in its ability to resolve the source space without prior hemodynamic and leadfield estimation. The two CNNs, one for EEG-to-fMRI and the other fMRI-to-EEG transcoding, are coupled in their source space representations, and given their architecture are able to capture both linear and non-linear transformations that map two imaging modalities into a common neural source space. We present results on simulated simultaneously acquired EEG-fMRI data and show the performance of mapping each modality to the other, the accuracy of recovering source space, and the effects of noise and variation in the simulated acquisition parameters, such as MRI slice timing, on the results.