Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals

Abstract Affective computing is an emerging multidisciplinary research field that is increasingly drawing the attention of researchers and practitioners in various fields, including artificial intelligence, natural language processing, cognitive and social sciences. Research in affective computing includes areas such as sentiment, emotion, and opinion modelling. The internet is an excellent source of data required for sentiment analysis, such as customer reviews of products, social media, forums, blogs, etc. Most of these data, called big data, are unstructured and unorganized. Hence there is a strong demand for developing suitable data processing techniques to process these rich and valuable data to produce useful information. Early surveys on sentiment and emotion recognition in the literature have been limited to discussions using text, audio, and visual modalities. So far, to the author's knowledge, a comprehensive survey combining physiological modalities with these other modalities for affective computing has yet to be reported. The objective of this paper is to fill the gap in this surveyed area. The usage of physiological modalities for affective computing brings several benefits in that the signals can be used in different environmental conditions, more robust systems can be constructed in combination with other modalities, and it has increased anti-spoofing characteristics. The paper includes extensive reviews on different frameworks and categories for state-of-the-art techniques, critical analysis of their performances, and discussions of their applications, trends and future directions to serve as guidelines for readers towards this emerging research area.