From the Inside Out: A Literature Review on Possibilities of Mobile Emotion Measurement and Recognition

Information systems are becoming increasingly intelligent and emotion artificial intelligence is an important component for the future. Therefore, the measurement and recognition of emotions is necessary and crucial. This paper presents a state of the art in the research field of mobile emotion measurement and recognition. The aim of this structured literature analysis using the PRISMA statement is to collect and classify the relevant literature and to provide an overview of the current status of mobile emotion recording and its future trends. A total of 59 articles were identified in the relevant literature databases, which can be divided into four main categories of emotion measurement. There was an increase of publications over the years in all four categories, but with a particularly strong increase in the areas of optical and vital-data-based recording. Over time, both the speed as well as the accuracy of the measurement has improved considerably in all four

[1]  Jason Williams,et al.  Emotion Recognition Using Bio-sensors: First Steps towards an Automatic System , 2004, ADS.

[2]  Moonhyun Kim,et al.  Mobile App Classification Method Using Machine Learning Based User Emotion Recognition , 2018, ICICM '18.

[3]  Benoit Huet,et al.  Evidence Theory-Based Multimodal Emotion Recognition , 2009, MMM.

[4]  Wei Gao,et al.  MoodMagician: a pervasive and unobtrusive emotion sensing system using mobile phones for improving human mental health , 2014, SenSys.

[5]  G. Bieber,et al.  Emotion Recognition on the Go : Providing Personalized Services Based on Emotional States , 2009 .

[6]  J. Russell.  A circumplex model of affect. , 1980 .

[7]  Robert Li Kam Wa.  MoodScope: Building a Mood Sensor from Smartphone Usage Patterns , 2012 .

[8]  Areej Al-Wabil,et al.  Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review , 2017 .

[9]  Nazia Hossain,et al.  EmoVoice: Finding My Mood from My Voice Signal , 2018, UbiComp/ISWC Adjunct.

[10]  Kai Kunze,et al.  Evaluation of Facial Expression Recognition by a Smart Eyewear for Facial Direction Changes, Repeatability, and Positional Drift , 2017, ACM Trans. Interact. Intell. Syst..

[11]  Anja Bachmann,et al.  Leveraging smartwatches for unobtrusive mobile ambulatory mood assessment , 2015, UbiComp/ISWC Adjunct.

[12]  Pradipta De,et al.  Evaluating effectiveness of smartphone typing as an indicator of user emotion , 2017, 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII).

[13]  Cecilia Mascolo,et al.  EmotionSense: a mobile phones based adaptive platform for experimental social psychology research , 2010, UbiComp.

[14]  Suja Palaniswamy,et al.  Emotion Recognition from Facial Expressions using Images with Pose, Illumination and Age Variation for Human-Computer/Robot Interaction , 2018 .

[15]  Stefan Winkler,et al.  ASCERTAIN: Emotion and Personality Recognition Using Commercial Sensors , 2018, IEEE Transactions on Affective Computing.

[16]  N. Frijda.  Moods, emotion episodes, and emotions. , 1993 .

[17]  Yue Zhang,et al.  Real-Time Emotion Detection via E-See , 2018, SenSys.

[18]  Eman M. G. Younis,et al.  Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection , 2019, Inf. Fusion.

[19]  Xu Chen,et al.  MoodExplorer , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[20]  Michael Cohen,et al.  "Emo Sim": expressing voice-based emotions in mobile interfaces , 2010, Humans and Computers.

[21]  J. Hagberg,et al.  The digitalization of retailing: an exploratory framework , 2016 .

[22]  Wei Zhang,et al.  Multimodal Emotion Recognition by extracting common and modality-specific information , 2018, SenSys.

[23]  Rajesh Krishna Balan,et al.  Demo: Towards Recognition of Rich Non-Negative Emotions Using Daily Wearable Devices , 2015, SenSys.

[24]  C. Vinola,et al.  A Survey on Human Emotion Recognition Approaches, Databases and Applications , 2015 .

[25]  Woonghee Lee,et al.  CEPP: Perceiving the Emotional State of the User Based on Body Posture , 2017 .

[26]  Elena Di Lascio,et al.  Unobtrusive Assessment of Students' Emotional Engagement during Lectures Using Electrodermal Activity Sensors , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[27]  Yu Cao,et al.  ReliefF-Based EEG Sensor Selection Methods for Emotion Recognition , 2016, Sensors.

[28]  Jung-Hsien Chiang,et al.  Portable Assessment of Emotional Status and Support System , 2014, HIS.

[29]  Jai Kyoung Sim,et al.  A Flexible and Wearable Human Stress Monitoring Patch , 2016, Scientific Reports.

[30]  Yuta Sugiura,et al.  Analysis of Multiple Users' Experience in Daily Life Using Wearable Device for Facial Expression Recognition , 2016, ACE.

[31]  Fadel Adib,et al.  Emotion recognition using wireless signals , 2016, MobiCom.

[32]  Sean M. Montgomery.  Measuring biological signals: concepts and practice , 2010, TEI '10.

[33]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[34]  R. Plutchik.  Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice , 2016 .

[35]  L. Zhu,et al.  Towards Mood Based Mobile Services and Applications , 2007, EuroSSC.

[36]  Shaun J. Canavan,et al.  Ubiquitous Emotion Recognition Using Audio and Video Data , 2018, UbiComp/ISWC Adjunct.

[37]  John A. Stankovic,et al.  Distant Emotion Recognition , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[38]  Jennifer Healey,et al.  Toward Machine Emotional Intelligence: Analysis of Affective Physiological State , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Manfred Tscheligi,et al.  Measuring Mobile Emotions: Measuring the Impossible? , 2009, Mobile HCI.

[40]  Michael Beigl,et al.  A wearable system for mood assessment considering smartphone features and data from mobile ECGs , 2016, UbiComp Adjunct.

[41]  Dzmitry Tsetserukou,et al.  World's first wearable humanoid robot that augments our emotions , 2010, AH.

[42]  John A. Stankovic,et al.  Real Time Distant Speech Emotion Recognition in Indoor Environments , 2017, MobiQuitous.

[43]  Kyu-Sik Park,et al.  A Study of Speech Emotion Recognition and Its Application to Mobile Services , 2007, UIC.

[44]  H. Lövheim.  A new three-dimensional model for emotions and monoamine neurotransmitters. , 2012, Medical hypotheses.

[45]  Lawrence K. Lam,et al.  A Flexible, Low Power, Compact, Mobile Sensor for Emotion Monitoring in Human Computer Interaction , 2018, HCI.

[46]  R. Amit,et al.  Value creation in E‐business , 2001 .

[47]  Hamid R. Tizhoosh,et al.  A comparative study of CNN, BoVW and LBP for classification of histopathological images , 2017, 2017 IEEE Symposium Series on Computational Intelligence (SSCI).

[48]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[49]  Kin K. Leung,et al.  Context-Awareness for Mobile Sensing: A Survey and Future Directions , 2016, IEEE Communications Surveys & Tutorials.

[50]  Kamlesh Mistry,et al.  Intelligent facial emotion recognition using moth-firefly optimization , 2016, Knowl. Based Syst..

[51]  Jari Multisilta,et al.  Social navigation with the collective mobile mood monitoring system , 2011, MindTrek.

[52]  Y. Lin,et al.  An Experimental Study on Physiological Parameters Toward Driver Emotion Recognition , 2007, HCI.

[53]  John A. Stankovic,et al.  SocialSense: A Collaborative Mobile Platform for Speaker and Mood Identification , 2015, EWSN.

[54]  Marta Blázquez,et al.  Fashion Shopping in Multichannel Retail: The Role of Technology in Enhancing the Customer Experience , 2014, Int. J. Electron. Commer..

[55]  Tsutomu Terada,et al.  A Lifelog System for Detecting Psychological Stress with Glass-equipped Temperature Sensors , 2016, AH.

[56]  M. Shamim Hossain,et al.  An Emotion Recognition System for Mobile Applications , 2017, IEEE Access.

[57]  Shaun J. Canavan,et al.  Ubiquitous Emotion Recognition with Multimodal Mobile Interfaces , 2018, UbiComp/ISWC Adjunct.

[58]  Mladen Russo,et al.  Wearable Emotion Recognition System based on GSR and PPG Signals , 2017, MMHealth@MM.

[59]  Marcel Brand,et al.  Automatische Emotionserkennung – Technologien, Deutung und Anwendungen , 2012, Informatik-Spektrum.

[60]  Jonathan Klein,et al.  Computers that recognise and respond to user emotion: theoretical and practical implications , 2002, Interact. Comput..

[61]  Nadine Mandran,et al.  Identifying emotions expressed by mobile users through 2D surface and 3D motion gestures , 2012, UbiComp '12.

[62]  Alberto L. Morán,et al.  Emotions Identification to Measure User Experience Using Brain Biometric Signals , 2015, HCI.

[63]  Christian Peter,et al.  Emotion in Human-Computer Interaction , 2012, Expanding the Frontiers of Visual Analytics and Visualization.

[64]  Wei Gao,et al.  Pervasive and unobtrusive emotion sensing for human mental health , 2013, 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops.

[65]  Pradipta De,et al.  Impact of experience sampling methods on tap pattern based emotion recognition , 2015, UbiComp/ISWC Adjunct.

[66]  Mirco Musolesi,et al.  Designing Effective Movement Digital Biomarkers for Unobtrusive Emotional State Mobile Monitoring , 2017, DigitalBioMarker@MobiSys.

[67]  A. Muaremi,et al.  Towards Measuring Stress with Smartphones and Wearable Devices During Workday and Sleep , 2013, BioNanoScience.

[68]  Yang Yan,et al.  Photoplethysmography based psychological stress detection with pulse rate variability feature differences and elastic net , 2018, Int. J. Distributed Sens. Networks.

[69]  Wei Zhang,et al.  Speech Emotion Recognition via Attention-based DNN from Multi-Task Learning , 2018, SenSys.