Toward Artificial Emotional Intelligence for Cooperative Social Human–Machine Interaction

The aptitude to identify the emotional states of others and response to exposed emotions is an important aspect of human social intelligence. Robots are expected to be prevalent in society to assist humans in various tasks. Human–robot interaction (HRI) is of critical importance in the assistive robotics sector. Smart digital assistants and assistive robots fail quite often when a request is not well defined verbally. When the assistant fails to provide services as desired, the person may exhibit an emotional response such as anger or frustration through expressions in their face and voice. It is critical that robots understand not only the language, but also human psychology. A novel affection-based perception architecture for cooperative HRIs is studied in this paper, where the agent is expected to recognize human emotional states, thus encourages a natural bonding between the human and the robotic artifact. We propose a method to close the loop using measured emotions to grade HRIs. This metric will be used as a reward mechanism to adjust the assistant’s behavior adaptively. Emotion levels from users are detected through vision and speech inputs processed by deep neural networks (NNs). Negative emotions exhibit a change in performance until the user is satisfied.

[1]  Jürgen Schmidhuber,et al.  Learning Precise Timing with LSTM Recurrent Networks , 2003, J. Mach. Learn. Res..

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Takeo Kanade,et al.  Recognizing Action Units for Facial Expression Analysis , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[7]  Mario Vieira,et al.  Towards the Development of Affective Facial Expression Recognition for Human-Robot Interaction , 2017, PETRA.

[8]  Berat A. Erol,et al.  Cloud-based control and vSLAM through cooperative Mapping and Localization , 2016, 2016 World Automation Congress (WAC).

[9]  Berat A. Erol,et al.  A deep vision landmark framework for robot navigation , 2017, 2017 12th System of Systems Engineering Conference (SoSE).

[10]  Maria E. Jabon,et al.  Real-time classification of evoked emotions using facial feature tracking and physiological responses , 2008, Int. J. Hum. Comput. Stud..

[11]  Paul Rad,et al.  Geospatial Event Detection by Grouping Emotion Contagion in Social Media , 2020, IEEE Transactions on Big Data.

[12]  Timnit Gebru,et al.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification , 2018, FAT.

[13]  Sotiris Makris,et al.  Human–robot interaction review and challenges on task planning and programming , 2016, Int. J. Comput. Integr. Manuf..

[14]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[15]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[17]  Dan R. Olsen,et al.  Metrics for Evaluating Human-Robot Interactions , 2003 .

[18]  Timothy F. Cootes,et al.  Active Appearance Models , 1998, ECCV.

[19]  Montse Pardàs,et al.  Emotion recognition based on MPEG-4 Facial Animation Parameters , 2002, 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[20]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[21]  Nicu Sebe,et al.  Multimodal approaches for emotion recognition: a survey , 2005, IS&T/SPIE Electronic Imaging.

[22]  Carlos D. Castillo,et al.  An All-In-One Convolutional Neural Network for Face Analysis , 2016, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[23]  Berat A. Erol,et al.  Improved Deep Neural Network Object Tracking System for Applications in Home Robotics , 2018 .

[24]  Yale Song,et al.  Multimodal human behavior analysis: learning correlation and interaction across modalities , 2012, ICMI '12.

[25]  Pierre-Yves Oudeyer,et al.  The poppy humanoid robot: Leg design for biped locomotion , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[26]  Michael A. Goodrich,et al.  Human-Robot Interaction: A Survey , 2008, Found. Trends Hum. Comput. Interact..

[27]  Jenay M. Beer,et al.  Toward a framework for levels of robot autonomy in human-robot interaction , 2014, Journal of human-robot interaction.

[28]  Afshin Dehghan,et al.  DAGER: Deep Age, Gender and Emotion Recognition Using Convolutional Neural Network , 2017, ArXiv.

[29]  Takeo Kanade,et al.  Detection, tracking, and classification of action units in facial expression , 2000, Robotics Auton. Syst..

[30]  Sethuraman Panchanathan,et al.  Multimodal emotion recognition using deep learning architectures , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[31]  Tal Hassner,et al.  Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns , 2015, ICMI.

[32]  Patrick Benavidez,et al.  Cloud-based realtime robotic Visual SLAM , 2015, 2015 Annual IEEE Systems Conference (SysCon) Proceedings.

[33]  Alessandro Gasparetto,et al.  System and method for recognizing human emotion state based on analysis of speech and facial feature extraction; applications to human-robot interaction , 2016, 2016 4th International Conference on Robotics and Mechatronics (ICROM).

[34]  Berat A. Erol,et al.  Speaker Recognition for Robotic Control via an IoT Device , 2018, 2018 World Automation Congress (WAC).

[35]  Jean Scholtz,et al.  Evaluation of human-robot interaction awareness in search and rescue , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[36]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[37]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[38]  Kerstin Dautenhahn,et al.  Socially intelligent robots: dimensions of human–robot interaction , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[39]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Michael J. Lyons,et al.  Coding facial expressions with Gabor wavelets , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.