Multi-Modal Physiological Data Fusion for Affect Estimation Using Deep Learning

Automated momentary estimation of positive and negative affects (PA and NA), the basic sense of feeling, can play an essential role in detecting the early signs of mood disorders. Physiological wearable sensors and machine learning have a potential to make such automated and continuous measurements. However, the physiological signals’ features that are associated with the subject-reported PA or NA may not be known. In this work, we use data-driven feature extraction based on deep learning to investigate the application of raw physiological signals for estimating PA and NA. Specifically, we propose two multi-modal data fusion methods with deep Convolutional Neural Networks. We use the proposed architecture to estimate PA and NA and also classify baseline, stress, and amusement emotions. The training and evaluation of the methods are performed using four physiological and one chest motion signal modalities collected using a chest sensing unit from 15 subjects. Overall, our proposed model performed better than traditional machine learning on hand-crafted features. Utilizing only two modalities, our proposed model estimated PA with a correlation of 0.69 (<inline-formula> <tex-math notation="LaTeX">${p} < 0.05$ </tex-math></inline-formula>) vs. 0.59 (<inline-formula> <tex-math notation="LaTeX">${p} < 0.05$ </tex-math></inline-formula>) with traditional machine learning. These correlations were 0.79 (<inline-formula> <tex-math notation="LaTeX">${p} < 0.05$ </tex-math></inline-formula>) vs. 0.73 (<inline-formula> <tex-math notation="LaTeX">${p} < 0.05$ </tex-math></inline-formula>) for NA estimation. The best emotion classification was achieved by the traditional method with 79% F1-score and 80% accuracy when all the four physiological modalities are used. This is while with only two modalities, the deep learning achieved 78% F1-score and 79% accuracy.