Recognizing Emotional Body Language Displayed by a Human-like Social Robot

Natural social human–robot interactions (HRIs) require that robots have the ability to perceive and identify complex human social behaviors and, in turn, be able to also display their own behaviors using similar communication modes. Recently, it has been found that body language plays an important role in conveying information about changes in human emotions during human–human interactions. Our work focuses on extending this concept to robotic affective communication during social HRI. Namely, in this paper, we explore the design of emotional body language for our human-like social robot, Brian 2.0. We develop emotional body language for the robot using a variety of body postures and movements identified in human emotion research. To date, only a handful of researchers have focused on the use of robotic body language to display emotions, with a significant emphasis being on the display of emotions through dance. Such emotional dance can be effective for small robots with large workspaces, however, it is not as appropriate for life-sized robots such as Brian 2.0 engaging in one-on-one interpersonal social interactions with a person. Experiments are presented to evaluate the feasibility of the robot’s emotional body language based on human recognition rates. Furthermore, a unique comparison study is presented to investigate the perception of human body language features displayed by the robot with respect to the same body language features displayed by a human actor.

[1]  H. Meeren,et al.  Rapid perceptual integration of facial expression and emotional body language. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[2]  Hiroshi Mizoguchi,et al.  Realization of Expressive Mobile Robot , 1997, Proceedings of International Conference on Robotics and Automation.

[3]  K. Barrett,et al.  Emotion communication and the development of the social emotions. , 1997, New directions for child development.

[4]  Beat Fasel,et al.  Automatic facial expression analysis: a survey , 2003, Pattern Recognit..

[5]  Marek P. Michalowski,et al.  Keepon : A Playful Robot for Research, Therapy, and Entertainment (Original Paper) , 2009 .

[6]  B. de Gelder,et al.  Body expressions influence recognition of emotions in the face and voice. , 2007, Emotion.

[7]  L. Wheeler,et al.  Review of personality and social psychology , 1980 .

[8]  Elisabeth André,et al.  Creation and Evaluation of emotion expression with body movement, sound and eye color for humanoid robots , 2011, 2011 RO-MAN.

[9]  M. D. Meijer.  The contribution of general features of body movement to the attribution of emotions , 1989 .

[10]  Matthew J. Hertenstein,et al.  Nonverbal channel use in communication of emotion: how may depend on why. , 2011, Emotion.

[11]  H. Wallbott.  Bodily expression of emotion , 1998 .

[12]  Khalid Aziz.  The key to perfect presentations , 1998 .

[13]  Mark Grimshaw,et al.  Facial expression of emotion and perception of the Uncanny Valley in virtual characters , 2011, Comput. Hum. Behav..

[14]  B. Parkinson.  Emotions are social. , 1996, British journal of psychology.

[15]  Cynthia Breazeal,et al.  Social interactions in HRI: the robot view , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[16]  A. Young,et al.  Emotion Perception from Dynamic and Static Body Expressions in Point-Light and Full-Light Displays , 2004, Perception.

[17]  Dong-Soo Kwon,et al.  Design of a robot head with arm-type antennae for emotional expression , 2007, 2007 International Conference on Control, Automation and Systems.

[18]  Masafumi Hashimoto,et al.  Remarks on designing of emotional movement for simple communication robot , 2010, 2010 IEEE International Conference on Industrial Technology.

[19]  Hillary Anger Elfenbein,et al.  Cross-cultural patterns in emotion recognition: highlighting design and analytical techniques. , 2002, Emotion.

[20]  Goldie Nejat,et al.  The Design of an Interactive Socially Assistive Robot for Patient Care , 2007 .

[21]  Hillary Anger Elfenbein,et al.  Cross-cultural patterns in emotion recognition: highlighting design and analytical techniques. , 2002 .

[22]  K. Scherer,et al.  Emotion expression in body action and posture. , 2012, Emotion.

[23]  Beatrice de Gelder,et al.  The Bodily Expressive Action Stimulus Test (BEAST). Construction and validation of a stimulus basis for measuring perception of whole body expression of emotions , 2011 .

[24]  C. Baird,et al.  The pilot study. , 2000, Orthopedic nursing.

[25]  Megumi Masuda,et al.  Emotion Detection from Body Motion of Human Form Robot Based on Laban Movement Analysis , 2009, PRIMA.

[26]  A. Takanishi,et al.  Various emotional expressions with emotion expression humanoid robot WE-4RII , 2004, IEEE Conference on Robotics and Automation, 2004. TExCRA Technical Exhibition Based..

[27]  Mark H. Chignell,et al.  Communication of Emotion in Social Robots through Simple Head and Arm Movements , 2011, Int. J. Soc. Robotics.

[28]  J. Graham,et al.  A Cross-cultural Study of the Communication of Emotion by Facial and Gestural Cues. , 1975 .

[29]  Goldie Nejat,et al.  A Socially Assistive Robot That Can Interpret Body Language , 2011 .

[30]  Maurizio Mancini,et al.  Expressive Copying Behavior for Social Agents: A Perceptual Analysis , 2012, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[31]  S. Brownlow,et al.  Perception of movement and dancer characteristics from point-light displays of dance , 1997 .

[32]  Tatsuya Nomura,et al.  Comparison on Identification of Affective Body Motions by Robots Between Elder People and University Students: A Case Study in Japan , 2010, Int. J. Soc. Robotics.

[33]  P Bull,et al.  The interpretation of posture through an alternative methodology to role play. , 1978, The British journal of social and clinical psychology.

[34]  J. Montepare,et al.  The Use of Body Movements and Gestures as Cues to Emotions in Younger and Older Adults , 1999 .

[35]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2009, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Armin Bruderlin,et al.  Perceiving affect from arm movement , 2001, Cognition.

[37]  Tzvi Ganel,et al.  Revisiting the relationship between the processing of gaze direction and the processing of facial expression. , 2011, Journal of experimental psychology. Human perception and performance.

[38]  Tomomasa Sato,et al.  Analysis of Impression of Robot Bodily Expression , 2002, J. Robotics Mechatronics.

[39]  F. Tanaka,et al.  Dance interaction with QRIO: a case study for non-boring interaction by using an entrainment ensemble model , 2004, RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759).

[40]  Hiroshi Kobayashi,et al.  Dynamic display of facial expressions on the face robot made by using a life mask , 2008, Humanoids 2008 - 8th IEEE-RAS International Conference on Humanoid Robots.

[41]  M. Bindemann,et al.  How do eye gaze and facial expression interact? , 2008 .

[42]  T. Hummel,et al.  “Taste Strips” – A rapid, lateralized, gustatory bedside identification test based on impregnated filter papers , 2009, Journal of Neurology.

[43]  C. Darwin.  The Expression of the Emotions in Man and Animals , .

[44]  P. Ekman.  Darwin, Deception, and Facial Expression , 2003, Annals of the New York Academy of Sciences.

[45]  J. Russell.  A circumplex model of affect. , 1980 .

[46]  Karen Caplovitz Barrett,et al.  The communication of emotion : current research from diverse perspectives , 1997 .

[47]  J. N. Bassili.  Facial motion in the perception of faces and of emotional expression. , 1978, Journal of experimental psychology. Human perception and performance.

[48]  Goldie Nejat,et al.  Human Affective State Recognition and Classification During Human-Robot Interaction Scenarios , 2009 .

[49]  J. Montepare,et al.  The identification of emotions from gait information , 1987 .

[50]  Goldie Nejat,et al.  Can I be of assistance? The intelligence behind an assistive robot , 2008, 2008 IEEE International Conference on Robotics and Automation.

[51]  Catherine Pelachaud,et al.  Modelling multimodal expression of emotion in a virtual agent , 2009, Philosophical Transactions of the Royal Society B: Biological Sciences.

[52]  A. Mehrabian.  Significance of posture and posiion in the communication of attitude and status relationships. , 1969, Psychological bulletin.

[53]  Ehud Sharlin,et al.  Playing Games with Robots - A Method for Evaluating Human-Robot Interaction , 2007 .

[54]  William Thomas James,et al.  A Study of the Expression of Bodily Posture , 1932 .

[55]  Iain R. Murray,et al.  Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. , 1993, The Journal of the Acoustical Society of America.

[56]  Hiroshi Ishiguro,et al.  Evaluating facial displays of emotion for the android robot Geminoid F , 2011, 2011 IEEE Workshop on Affective Computational Intelligence (WACI).

[57]  P. Silvia.  Interest—The Curious Emotion , 2008 .

[58]  M. G. Preston,et al.  On certain conditions controlling the realism and irrealism of aspirations. , 1947, Journal of experimental psychology.

[59]  Daniela M. Romano,et al.  Perception of Emotions from Static Postures , 2007, ACII.

[60]  P SumathiC.,et al.  Automatic Facial Expression Analysis A Survey , 2012 .

[61]  Mark C. Coulson.  Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence , 2004 .

[62]  Zhe Zhang,et al.  An Intelligent Socially Assistive Robot for Health Care , 2008 .

[63]  A. Takanishi,et al.  Development of a new human-like head robot WE-4 , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[64]  Shlomo Hareli,et al.  What's Social About Social Emotions? , 2008 .

[65]  R. Laban.  The mastery of movement on the stage , 1950 .

[66]  K. Mogg,et al.  Some methodological issues in assessing attentional biases for threatening faces in anxiety: a replication study using a modified version of the probe detection task. , 1999, Behaviour research and therapy.

[67]  Megumi Masuda,et al.  Motion rendering system for emotion expression of human form robots based on Laban movement analysis , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[68]  Eyal Amir,et al.  Human—Android Interaction in the Near and Distant Future , 2009, Perspectives on psychological science : a journal of the Association for Psychological Science.

[69]  J. Barbalet,et al.  Boredom and social meaning. , 1999, The British journal of sociology.

[70]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[71]  R. Gur,et al.  Differences in facial expressions of four universal emotions , 2004, Psychiatry Research.

[72]  Zhe Zhang,et al.  Human Body Pose Interpretation and Classification for Social Human-Robot Interaction , 2011, Int. J. Soc. Robotics.

[73]  Jung-Jin Yang.  Principles of Practice in Multi-Agent Systems, 12th International Conference, PRIMA 2009, Nagoya, Japan, December 14-16, 2009. Proceedings , 2009, PRIMA.

[74]  J. Hoogstraten,et al.  Head position and spinal position as determinants of perceived emotional state. , 1995, Perceptual and motor skills.

[75]  Aryel Beck,et al.  Towards an Affect Space for robots to display emotional body language , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[76]  S. R. St. J. Neill,et al.  Children's Reported Responses to Teachers' Non-verbal Signals: a pilot study , 1986 .

[77]  P. Ekman,et al.  Head and body cues in the judgment of emotion: a reformulation. , 1967, Perceptual and motor skills.

[78]  R. D. Walk,et al.  Perception of emotion from body posture , 1986 .

[79]  Keith Oatley,et al.  The Intentional and Social Nature of Human Emotions: Reconsideration of the Distinction Between Basic and Non‐basic Emotions , 1996 .

[80]  Catherine L Reed,et al.  Relative contributions of face and body configurations: Perceiving emotional state and motion intention , 2012, Cognition & emotion.

[81]  P. Shaver,et al.  Cross-cultural similarities and differences in emotion and its representation. , 1992 .