Hybrid spatiotemporal models for sentiment classification via galvanic skin response

Abstract Sentiment plays an important role in cognition, creativity, attention, and decision making in people’s daily lives. Researchers have made great progress in sentiment recognition through images and speech. In this paper, a multimodal dataset is proposed for sentiment classification (MDSTC 1 )—a multimodal dataset collected with multimodal channels by customized physiological sensors and manually labelled for human sentiment analysis. MDSTC contains galvanic skin response (GSR), pulse, speech, and facial expression data of 100 volunteers labelled with timeline, emotional self-ratings, and personal information such as age, sex and Big Five personality scores, which were collected while watching video clips to stimulate emotions. Using the MDSTC dataset, the GSR channel with six types of emotion labels is used to perform human sentiment analysis in this paper. The GSR signal is converted into a spectrogram to adopt image-related methods and deep learning models. A convolutional neural network, long short-term memory, and self-attention mechanism are adopted and combined, and spatiotemporal hybrid models are proposed to perform human sentiment analysis. Comparisons are performed between the proposed model and state-of-the-art models in related works. The experimental results show that with the proposed spatiotemporal hybrid models, better results are obtained with respect to precision, recall, and F1-score. The experimental results also show that with the proposed spatiotemporal hybrid models working with GSR, people’s emotional changes can be obtained in real time with high precision.