Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives

A fascinating challenge in the field of human–robot interaction is the possibility to endow robots with emotional intelligence in order to make the interaction more intuitive, genuine, and natural. To achieve this, a critical point is the capability of the robot to infer and interpret human emotions. Emotion recognition has been widely explored in the broader fields of human–machine interaction and affective computing. Here, we report recent advances in emotion recognition, with particular regard to the human–robot interaction context. Our aim is to review the state of the art of currently adopted emotional models, interaction modalities, and classification strategies and offer our point of view on future developments and critical issues. We focus on facial expressions, body poses and kinematics, voice, brain activity, and peripheral physiological responses, also providing a list of available datasets containing data from these modalities.

[1]  Jinkyu Lee,et al.  High-level feature representation using recurrent neural network for speech emotion recognition , 2015, INTERSPEECH.

[2]  Lola Cañamero,et al.  Emotion understanding from the perspective of autonomous robots research , 2005, Neural Networks.

[3]  Torbjørn S. Dahl,et al.  How does the robot feel? Perception of valence and arousal in emotional body language , 2018, Paladyn J. Behav. Robotics.

[4]  Aditya Inamdar,et al.  A review of recent trends in EEG based Brain-Computer Interface , 2019, 2019 International Conference on Computational Intelligence in Data Science (ICCIDS).

[5]  Bin Hu,et al.  Feature-level fusion of multimodal physiological signals for emotion recognition , 2015, 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[6]  Massimiliano Palmiero,et al.  Frontal EEG Asymmetry of Mood: A Mini-Review , 2017, Front. Behav. Neurosci..

[7]  Vincent G. Duffy,et al.  The Effect of a Humanoid Robot’s Emotional Behaviors on Users’ Emotional Responses: Evidence from Pupillometry and Electroencephalography Measures , 2019, Int. J. Hum. Comput. Interact..

[8]  Mehdi Ghayoumi A Quick Review of Deep Learning in Facial Expression , 2017 .

[9]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Li Zhang,et al.  Adaptive 3D facial action intensity estimation and emotion recognition , 2015, Expert Syst. Appl..

[11]  Thierry Pun,et al.  A channel selection method for EEG classification in emotion assessment based on synchronization likelihood , 2007, 2007 15th European Signal Processing Conference.

[12]  Balamurugan Balusamy,et al.  Wearable Devices and IoT , 2019, A Handbook of Internet of Things in Biomedical and Cyber Physical System.

[13]  Mario Vieira,et al.  Towards the Development of Affective Facial Expression Recognition for Human-Robot Interaction , 2017, PETRA.

[14]  Wai-Chi Fang,et al.  Design of Intelligent EEG System for Human Emotion Recognition with Convolutional Neural Network , 2019, 2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS).

[15]  Arvid Kappas,et al.  Smile When You Read This, Whether You Like It or Not: Conceptual Challenges to Affect Detection , 2010, IEEE Transactions on Affective Computing.

[16]  Sergio Escalera,et al.  Survey on Emotional Body Gesture Recognition , 2018, IEEE Transactions on Affective Computing.

[17]  Kjell Elenius,et al.  Emotion Recognition , 2009, Computers in the Human Interaction Loop.

[18]  R. Gunderman,et al.  Emotional intelligence. , 2011, Journal of the American College of Radiology : JACR.

[19]  Meng Xia,et al.  Estimating Emotional Intensity from Body Poses for Human-Robot Interaction , 2019, ArXiv.

[20]  Petros Maragos,et al.  Fusing Body Posture With Facial Expressions for Joint Recognition of Affect in Child–Robot Interaction , 2019, IEEE Robotics and Automation Letters.

[21]  Wanqing Li,et al.  Human detection from images and videos: A survey , 2016, Pattern Recognit..

[22]  Daniele Mazzei,et al.  Development and testing of a multimodal acquisition platform for human-robot interaction affective studies , 2014, J. Hum. Robot Interact..

[23]  Ting Zhang,et al.  Facial Expression Recognition Based on Deep Learning: A Survey , 2017 .

[24]  Danyang Li,et al.  Random Deep Belief Networks for Recognizing Emotions from Speech Signals , 2017, Comput. Intell. Neurosci..

[25]  Beno Benhabib,et al.  A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI , 2016, J. Intell. Robotic Syst..

[26]  Yongzhao Zhan,et al.  Using Kinect for real-time emotion recognition via facial expressions , 2015, Frontiers of Information Technology & Electronic Engineering.

[27]  Mohammad Soleymani,et al.  A Multimodal Database for Affect Recognition and Implicit Tagging , 2012, IEEE Transactions on Affective Computing.

[28]  Paolo Dario,et al.  Emotion Modelling for Social Robotics Applications: A Review , 2018 .

[29]  Bayya Yegnanarayana,et al.  Analysis of Emotional Speech - A Review , 2016, Toward Robotic Socially Believable Behaving Systems.

[30]  Arcangelo Merla,et al.  Thermal Infrared Imaging-Based Affective Computing and Its Application to Facilitate Human Robot Interaction: A Review , 2020 .

[31]  Beno Benhabib,et al.  You Are Doing Great! Only One Rep Left: An Affect-Aware Social Robot for Exercising , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[32]  Andrzej Cichocki,et al.  Individual Classification of Emotions Using EEG , 2014 .

[33]  Heinrich H. Bülthoff,et al.  The MPI Emotional Body Expressions Database for Narrative Scenarios , 2014, PloS one.

[34]  Eva Cerezo,et al.  Designing emotional BDI agents: good practices and open questions , 2019, The Knowledge Engineering Review.

[35]  Stefano Federici,et al.  Extracting neurophysiological signals reflecting users' emotional and affective responses to BCI use: A systematic literature review. , 2015, NeuroRehabilitation.

[36]  Heather C. Lench,et al.  Discrete emotions predict changes in cognition, judgment, experience, behavior, and physiology: a meta-analysis of experimental emotion elicitations. , 2011, Psychological bulletin.

[37]  Luigi Cinque,et al.  Towards EEG-based BCI driven by emotions for addressing BCI-Illiteracy: a meta-analytic review , 2018, Behav. Inf. Technol..

[38]  P. Ekman An argument for basic emotions , 1992 .

[39]  Ioannis A. Kakadiaris,et al.  Estimating anthropometry and pose from a single image , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[40]  Rytis Maskeliunas,et al.  Distributed under Creative Commons Cc-by 4.0 Consumer-grade Eeg Devices: Are They Usable for Control Tasks? , 2022 .

[41]  Sonya S. Kwak,et al.  What makes people empathize with an emotional robot?: The impact of agency and physical embodiment on human empathy for a robot , 2013, 2013 IEEE RO-MAN.

[42]  Haizhou Li,et al.  Acoustic emotion recognition based on fusion of multiple feature-dependent deep Boltzmann machines , 2014, The 9th International Symposium on Chinese Spoken Language Processing.

[43]  Paul A. Viola,et al.  Detecting Pedestrians Using Patterns of Motion and Appearance , 2005, International Journal of Computer Vision.

[44]  Eva Hudlicka,et al.  Guidelines for Designing Computational Models of Emotions , 2011, Int. J. Synth. Emot..

[45]  S. A. Hosseini,et al.  Higher Order Spectra Analysis of EEG Signals in Emotional Stress States , 2010, 2010 Second International Conference on Information Technology and Computer Science.

[46]  Martin Buss,et al.  Feature Extraction and Selection for Emotion Recognition from EEG , 2014, IEEE Transactions on Affective Computing.

[47]  Jayashree V. Patil,et al.  Real time facial expression recognition using RealSense camera and ANN , 2016, 2016 International Conference on Inventive Computation Technologies (ICICT).

[48]  Adriana Tapus,et al.  Interactive Robot Learning for Multimodal Emotion Recognition , 2019, ICSR.

[49]  Areej Al-Wabil,et al.  Review and Classification of Emotion Recognition Based on EEG Brain-Computer Interface System Research: A Systematic Review , 2017 .

[50]  B. Prabhakaran,et al.  Real-Time Mobile Facial Expression Recognition System -- A Case Study , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[51]  Kristen A. Lindquist,et al.  The hundred-year emotion war: are emotions natural kinds or psychological constructions? Comment on Lench, Flores, and Bench (2011). , 2013, Psychological bulletin.

[52]  Thierry Dutoit,et al.  Performance of the Emotiv Epoc headset for P300-based applications , 2013, Biomedical engineering online.

[53]  Sergio Escalera,et al.  Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Beno Benhabib,et al.  User Affect Elicitation with a Socially Emotional Robot , 2020, Robotics.

[55]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[56]  Stefan Wermter,et al.  Emotional expression recognition with a cross-channel convolutional neural network for human-robot interaction , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[57]  M. Sams,et al.  Distributed affective space represents multiple emotion categories across the human brain , 2018, Social cognitive and affective neuroscience.

[58]  Min Wu,et al.  A multimodal emotional communication based humans-robots interaction system , 2016, 2016 35th Chinese Control Conference (CCC).

[59]  Zhen-Tao Liu,et al.  Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection , 2019, IEEE Transactions on Cognitive and Developmental Systems.

[60]  Seungryong Kim,et al.  Context-Aware Emotion Recognition Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[61]  Martin Kuška,et al.  Modeling Semantic Emotion Space Using a 3D Hypercube-Projection: An Innovative Analytical Approach for the Psychology of Emotions , 2016, Front. Psychol..

[62]  Sylvia D. Kreibig,et al.  Autonomic nervous system activity in emotion: A review , 2010, Biological Psychology.

[63]  Mariusz Szwoch,et al.  Facial emotion recognition using depth data , 2015, 2015 8th International Conference on Human System Interaction (HSI).

[64]  Yoshua Bengio,et al.  Challenges in representation learning: A report on three machine learning contests , 2013, Neural Networks.

[65]  Qiang Ji,et al.  Emotion recognition from thermal infrared images using deep Boltzmann machine , 2014, Frontiers of Computer Science.

[66]  Goldie Nejat,et al.  How Robots Influence Humans: A Survey of Nonverbal Communication in Social Human–Robot Interaction , 2019, International Journal of Social Robotics.

[67]  Lola Cañamero,et al.  Embodied Robot Models for Interdisciplinary Emotion Research , 2019, IEEE Transactions on Affective Computing.

[68]  Carlos M. Oppus,et al.  Face Detection and Recognition of the Seven Emotions via Facial Expression: Integration of Machine Learning Algorithm into the NAO Robot , 2020, 2020 5th International Conference on Control and Robotics Engineering (ICCRE).

[69]  Moriel Zelikowsky,et al.  Design of a Neurally Plausible Model of Fear Learning , 2011, Front. Behav. Neurosci..

[70]  Bao-Liang Lu,et al.  Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks , 2015, IEEE Transactions on Autonomous Mental Development.

[71]  Norsinnira Zainul Azlan,et al.  Deep Learning Methods for Facial Expression Recognition , 2019, 2019 7th International Conference on Mechatronics Engineering (ICOM).

[72]  Jennifer G. Dy,et al.  Emotion Fingerprints or Emotion Populations? A Meta-Analytic Investigation of Autonomic Features of Emotion Categories , 2018, Psychological bulletin.

[73]  Teodiano Freire Bastos Filho,et al.  Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction , 2019, Sensors.

[74]  Antonio Camurri,et al.  Toward a Minimal Representation of Affective Gestures , 2011, IEEE Transactions on Affective Computing.

[75]  Anton Nijholt,et al.  Usability of Three Electroencephalogram Headsets for Brain-Computer Interfaces: A Within Subject Comparison , 2015, Interact. Comput..

[76]  Denis Delisle-Rodriguez,et al.  Emotion analysis in children through facial emissivity of infrared thermal imaging , 2019, PloS one.

[77]  M. Akin,et al.  Comparison of Wavelet Transform and FFT Methods in the Analysis of EEG Signals , 2002, Journal of Medical Systems.

[78]  Jenn-Jier James Lien,et al.  AdaBoost Learning for Human Detection Based on Histograms of Oriented Gradients , 2007, ACCV.

[79]  Takeo Kanade,et al.  The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[80]  David A. Forsyth,et al.  Probabilistic Methods for Finding People , 2001, International Journal of Computer Vision.

[81]  Deepak Ghimire,et al.  Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines , 2013, Sensors.

[82]  Witold Pedrycz,et al.  Adaptive Feature Selection-Based AdaBoost-KNN With Direct Optimization for Dynamic Emotion Recognition in Human–Robot Interaction , 2019, IEEE Transactions on Emerging Topics in Computational Intelligence.

[83]  Stefan Wermter,et al.  A neurocomputational amygdala model of auditory fear conditioning: A hybrid system approach , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[84]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[85]  Philip A. Kragel,et al.  Decoding the Nature of Emotion in the Brain , 2016, Trends in Cognitive Sciences.

[86]  Emiliano Lorini,et al.  Computational Modeling of Emotion: Toward Improving the Inter- and Intradisciplinary Exchange , 2013, IEEE Transactions on Affective Computing.

[87]  Silvia Rossi,et al.  Emotional and Behavioural Distraction by a Social Robot for Children Anxiety Reduction During Vaccination , 2020, Int. J. Soc. Robotics.

[88]  Fakhri Karray,et al.  Survey on speech emotion recognition: Features, classification schemes, and databases , 2011, Pattern Recognit..

[89]  V. Gallese,et al.  Thermal infrared imaging in psychophysiology: Potentialities and limits , 2014, Psychophysiology.

[90]  A. Mehrabian Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in Temperament , 1996 .

[91]  Rosalind W. Picard Affective Computing for HCI , 1999, HCI.

[92]  T. Johnstone The effect of emotion on voice production and speech acoustics , 2017 .

[93]  Julieta Domínguez-Soberanes,et al.  Facial Emotion Recognition: A Comparison of Different Landmark-Based Classifiers , 2018, 2018 International Conference on Research in Intelligent and Computing in Engineering (RICE).

[94]  J. Russell A circumplex model of affect. , 1980 .

[95]  Stefan Wermter,et al.  Emotion Recognition from Body Expressions with a Neural Network Architecture , 2017, HAI.

[96]  Thamer Alhussain,et al.  Speech Emotion Recognition Using Deep Learning Techniques: A Review , 2019, IEEE Access.

[97]  Weihua Cao,et al.  Combining 2D Gabor and Local Binary Pattern for Facial Expression Recognition Using Extreme Learning Machine , 2019, J. Adv. Comput. Intell. Intell. Informatics.

[98]  Y. X. Zou,et al.  An experimental study of speech emotion recognition based on deep convolutional neural networks , 2015, 2015 International Conference on Affective Computing and Intelligent Interaction (ACII).

[99]  Nikolaos Mavridis,et al.  A review of verbal and non-verbal human-robot interactive communication , 2014, Robotics Auton. Syst..

[100]  Martina Ruocco,et al.  Better alone than in bad company , 2019 .

[101]  Min Wu,et al.  Dynamic Emotion Understanding in Human–Robot Interaction Based on Two-Layer Fuzzy SVR-TS Model , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[102]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[103]  Kin-Man Lam,et al.  Histogram-based local descriptors for facial expression recognition (FER): A comprehensive study , 2018, J. Vis. Commun. Image Represent..

[104]  Sriparna Saha,et al.  A study on emotion recognition from body gestures using Kinect sensor , 2014, 2014 International Conference on Communication and Signal Processing.

[105]  Mikel Val-Calvo,et al.  Affective Robot Story-Telling Human-Robot Interaction: Exploratory Real-Time Emotion Estimation Analysis Using Facial Expressions and Physiological Signals , 2020, IEEE Access.

[106]  Christopher Joseph Pal,et al.  Recurrent Neural Networks for Emotion Recognition in Video , 2015, ICMI.

[107]  Ioannis Patras,et al.  AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups , 2017, IEEE Transactions on Affective Computing.

[108]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[109]  Annamalai Manickavasagan,et al.  Thermal Infrared Imaging , 2014 .

[110]  Nadia Bianchi-Berthouze,et al.  Continuous Recognition of Player's Affective Body Expression as Dynamic Quality of Aesthetic Experience , 2012, IEEE Transactions on Computational Intelligence and AI in Games.

[111]  Umesh Chavan,et al.  Facial Expression Recognition- Review , 2013 .

[112]  Shangfei Wang,et al.  Emotion Recognition Using Hidden Markov Models from Facial Temperature Sequence , 2011, ACII.

[113]  Leontios J. Hadjileontiadis,et al.  Emotion Recognition From EEG Using Higher Order Crossings , 2010, IEEE Transactions on Information Technology in Biomedicine.

[114]  Soraia M. Alarcão,et al.  Emotions Recognition Using EEG Signals: A Survey , 2019, IEEE Transactions on Affective Computing.

[115]  Emily C. Collins,et al.  Presence of Life-Like Robot Expressions Influences Children’s Enjoyment of Human-Robot Interactions in the Field , 2015 .

[116]  ByoungChul Ko,et al.  A Brief Review of Facial Emotion Recognition Based on Visual Information , 2018, Sensors.

[117]  Brian Scassellati,et al.  A thermal emotion classifier for improved human-robot interaction , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[118]  Thierry Pun,et al.  DEAP: A Database for Emotion Analysis ;Using Physiological Signals , 2012, IEEE Transactions on Affective Computing.

[119]  Katarzyna Wac,et al.  Designing Emotionally Expressive Robots: A Comparative Study on the Perception of Communication Modalities , 2017, HAI.

[120]  Xiang Li,et al.  Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[121]  Jun-Wei Mao,et al.  Speech emotion recognition based on feature selection and extreme learning machine decision tree , 2018, Neurocomputing.

[122]  Felipe Trujillo-Romero,et al.  Multimodal emotion recognition with evolutionary computation for human-robot interaction , 2016, Expert Syst. Appl..

[123]  Jinhua She,et al.  Two-layer fuzzy multiple random forest for speech emotion recognition in human-robot interaction , 2020, Inf. Sci..