Dual-CNN based multi-modal sleep scoring with temporal correlation driven fine-tuning

Abstract Examination of dysfunctional brain dynamics often needs to tackle the difficulties of analyzing neural data of multiple modalities as recordings from a single source (such as EEG) do not always suffice in characterizing the brain states of interest. A typical example is sleep scoring from Polysomnography (PSG) for evaluation of sleep disorder, which has long been an onerous task and constantly results in unstable performance. Despite numerous successes achieved by the deep learning technologies recently booming along this direction, a grand challenge still remains: how to enable a stable solution to characterize the sleep stages (brain states) with temporal correlations embedded in the multi-modal recordings mainly from the brain activities without the support of sufficient a priori knowledge of the subjects. Aiming at the challenge, this study develops a deep learning framework for multi-modal sleep scoring. The framework uses a “dual-CNN” to simultaneously process inputs of two forms, i.e., temporal or time-frequency. The inputs of each branch may either consist of organized epochs of multiple modalities or their time-frequency representations. The fused deep features of the original multi-modal dataset will then be derived. The correlation of the original consecutive epochs embedded in the deep features can be reinforced through an RNN layer to aid classification with the final results fine-tuned by a customized Markov chain model, which considers the temporal correlations embedded in the deep features to amortize the potential bias resulted from the imbalance of brain states represented by the features. A case study of sleep scoring has been performed against a public Sleep-EDF dataset. Experimental results indicate that 1) the multi-modal features generated by the dual-CNN model help in significantly improve the accuracy of neural data classification; 2) the overall framework outperforms the state-of-the-art counterparts with an accuracy of 84.9% achieved in sleep scoring.