Multi-modal Fusion Methods for Robust Emotion Recognition using Body-worn Physiological Sensors in Mobile Environments

High-accuracy physiological emotion recognition typically requires participants to wear or attach obtrusive sensors (e.g., Electroencephalograph). To achieve precise emotion recognition using only wearable body-worn physiological sensors, my doctoral work focuses on researching and developing a robust sensor fusion system among different physiological sensors. Developing such fusion system has three problems: 1) how to pre-process signals with different temporal characteristics and noise models, 2) how to train the fusion system with limited labeled data and 3) how to fuse multiple signals with inaccurate and inexact ground truth. To overcome these challenges, I plan to explore semi-supervised, weakly supervised and unsupervised machine learning methods to obtain precise emotion recognition in mobile environments. By developing such techniques, we can measure the user engagement with larger amounts of participants and apply the emotion recognition techniques in a variety of scenarios such as mobile video watching and online education.

[1]  W. Cannon.  Again the James-Lange and the thalamic theories of emotion. , 1931 .

[2]  N. Birbaumer,et al.  The Structure of emotion : psychophysiological, cognitive, and clinical aspects , 1993 .

[3]  E. Cook,et al.  The structure of emotion: Psychophysiological, cognitive and clinical aspects , 1996 .

[4]  A. Mehrabian.  Comparison of the PAD and PANAS as models for describing emotions and for differentiating anxiety from depression , 1997 .

[5]  Carla E. Brodley,et al.  Identifying Mislabeled Training Data , 1999, J. Artif. Intell. Res..

[6]  Peter Robinson,et al.  Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[7]  M. Bradley,et al.  The pupil as a measure of emotional arousal and autonomic activation. , 2008, Psychophysiology.

[8]  James R. Foulds,et al.  A review of multi-instance learning assumptions , 2010, The Knowledge Engineering Review.

[9]  Rafael A. Calvo,et al.  Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications , 2010, IEEE Transactions on Affective Computing.

[10]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[11]  Zhi-Hua Zhou,et al.  Ensemble Methods: Foundations and Algorithms , 2012 .

[12]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[13]  Dirk Van,et al.  Ensemble Methods: Foundations and Algorithms , 2012 .

[14]  G. Band,et al.  Pupil dilation in the Simon task as a marker of conflict processing , 2013, Front. Hum. Neurosci..

[15]  Jeff A. Bilmes,et al.  Deep Canonical Correlation Analysis , 2013, ICML.

[16]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[17]  Olga Sourina,et al.  Real-Time Subject-Dependent EEG-Based Emotion Recognition Algorithm , 2014, Trans. Comput. Sci..

[18]  Emre Ertin,et al.  cStress: towards a gold standard for continuous stress assessment in the mobile environment , 2015, UbiComp.

[19]  C. Izard,et al.  Measures of affect dimensions , 2015 .

[20]  M. Shamim Hossain,et al.  An Emotion Recognition System for Mobile Applications , 2017, IEEE Access.

[21]  Zhi-Hua Zhou,et al.  A brief introduction to weakly supervised learning , 2018 .

[22]  S. Nagamani,et al.  A Mobile Cloud-Based Approach for Secure M-Health Prediction Application , 2018 .

[23]  Tianyi Zhang,et al.  How Old Do You Look? Inferring Your Age from Your Gaze , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[24]  Zhenqi Li,et al.  A Review of Emotion Recognition Using Physiological Signals , 2018, Sensors.

[25]  Bin Jiang,et al.  Data-driven and deep learning-based detection and diagnosis of incipient faults with application to electrical traction systems , 2020, Neurocomputing.

[26]  Bin Jiang,et al.  A Review of Fault Detection and Diagnosis for the Traction System in High-Speed Trains , 2020, IEEE Transactions on Intelligent Transportation Systems.