EmoChat: Bringing Multimodal Emotion Detection to Mobile Conversation

Online chatting is very popular nowadays. However, most of the chatting softwares are based on pure text messages, which cannot completely convey users' emotions, causing information asymmetry. In this paper, we propose a new online chatting system, named EmoChat, which automatically identifies the emotions of the users and attaches the identification result to the messages sent by the user, allowing users to know the emotions of each other during online chatting. EmoChat analyzes the real-time emotions of users based on a joint consideration of facial expressions and text messages. Specifically, we propose an information entropy based method to fuse the multimodal information of these two pieces of complementary information. Furthermore, by realizing the context-sensitive property of the emotion information, a Hidden Markov Model based method is proposed to improve the emotion recognition accuracy with the context information. We implement EmoChat and evaluate its performance through a series of experiments. The experimental results show that EmoChat achieves an accuracy of 76.25% for emotion polarity recognition and an accuracy of 51.64% for emotion category recognition. Moreover, the delay when sending a message with emotions attached is within 50ms on the mobile devices.

[1]  Rong Li,et al.  Privacy Leakage in Mobile Sensing: Your Unlock Passwords Can Be Leaked through Wireless Hotspot Functionality , 2016, Mob. Inf. Syst..

[2]  Yuan He,et al.  Smart-DJ: Context-Aware Personalization for Music Recommendation on Smartphones , 2016, 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS).

[3]  Junmo Kim,et al.  Joint Fine-Tuning in Deep Neural Networks for Facial Expression Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[4]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[5]  Ilia Uma.  physiological signals based human emotion recognition a review , 2014 .

[6]  Nicu Sebe,et al.  MULTIMODAL EMOTION RECOGNITION , 2005 .

[7]  Jacob Aron.  How innovative is Apple's new voice assistant, Siri? , 2011 .

[8]  Mohammad H. Mahoor,et al.  Going deeper in facial expression recognition using deep neural networks , 2015, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[9]  Yunhao Liu,et al.  Pervasive Floorplan Generation Based on Only Inertial Sensing: Feasibility, Design, and Implementation , 2017, IEEE Journal on Selected Areas in Communications.

[10]  Rudra Pratap Deb Nath,et al.  Emotion extraction from real time chat messenger , 2014, 2014 International Conference on Informatics, Electronics & Vision (ICIEV).

[11]  Shaogang Gong,et al.  Facial expression recognition based on Local Binary Patterns: A comprehensive study , 2009, Image Vis. Comput..

[12]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[13]  Matias Valdenegro-Toro,et al.  Real-time Convolutional Neural Networks for emotion and gender classification , 2017, ESANN.

[14]  P. Ekman.  An argument for basic emotions , 1992 .

[15]  Ginevra Castellano,et al.  Recognising Human Emotions from Body Movement and Gesture Dynamics , 2007, ACII.

[16]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[17]  Jiang Qian,et al.  Text sentiment analysis based on long short-term memory , 2016, 2016 First IEEE International Conference on Computer Communication and the Internet (ICCCI).

[18]  Yong Peng,et al.  EEG-based emotion classification using deep belief networks , 2014, 2014 IEEE International Conference on Multimedia and Expo (ICME).

[19]  Cha Zhang,et al.  Image based Static Facial Expression Recognition with Multiple Deep Network Learning , 2015, ICMI.

[20]  Mitsuru Ishizuka,et al.  Emotion Estimation and Reasoning Based on Affective Textual Interaction , 2005, ACII.

[21]  Yunhao Liu,et al.  Share Brings Benefits: Towards Maximizing Revenue for Crowdsourced Mobile Network Access , 2017, 2017 14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON).

[22]  Ying Zhang,et al.  Text Emotion Distribution Learning via Multi-Task Convolutional Neural Network , 2018, IJCAI.

[23]  Harry Shum,et al.  The Design and Implementation of XiaoIce, an Empathetic Social Chatbot , 2018, CL.

[24]  Hua Wang,et al.  Communicating emotions in online chat using physiological sensors and animated text , 2004, CHI EA '04.

[25]  Yoshua Bengio,et al.  Challenges in representation learning: A report on three machine learning contests , 2013, Neural Networks.

[26]  Jiucang Hao,et al.  Emotion recognition by speech signals , 2003, INTERSPEECH.

[27]  Francesca Odone,et al.  Real-time Automatic Emotion Recognition from Body Gestures , 2014, ArXiv.

[28]  Oh-Wook Kwon,et al.  EMOTION RECOGNITION BY SPEECH SIGNAL , 2003 .

[29]  Dana Kulic,et al.  Affective State Estimation for Human–Robot Interaction , 2007, IEEE Transactions on Robotics.

[30]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[31]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[32]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[33]  Xi Chen,et al.  Privacy-preserving high-quality map generation with participatory sensing , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[34]  Dong Yu,et al.  Speech emotion recognition using deep neural network and extreme learning machine , 2014, INTERSPEECH.