A model for synthesizing a combined verbal and nonverbal behavior based on personality traits in human-robot interaction

Robots are more and more present in our daily life; they have to move into human-centered environments, to interact with humans, and to obey some social rules so as to produce an appropriate social behavior in accordance with human's profile (i.e., personality, state of mood, and preferences). Recent researches discussed the effect of personality traits on the verbal and nonverbal production, which plays a major role in transferring and understanding messages in a social interaction between a human and a robot. The characteristics of the generated gestures (e.g., amplitude, direction, rate, and speed) during the nonverbal communication can differ according to the personality trait, which, similarly, influences the verbal content of the human speech in terms of verbosity, repetitions, etc. Therefore, our research tries to map a human's verbal behavior to a corresponding combined robot's verbal-nonverbal behavior based on the personality dimensions of the interacting human. The system estimates first the interacting human's personality traits through a psycholinguistic analysis of the spoken language, then it uses PERSONAGE natural language generator that tries to generate a corresponding verbal language to the estimated personality traits. Gestures are generated by using BEAT toolkit, which performs a linguistic and contextual analysis of the generated language relying on rules derived from extensive research into human conversational behavior. We explored the human-robot personality matching aspect and the differences of the adapted mixed robot's behavior (gesture and speech) over the adapted speech only robot's behavior in an interaction. Our model validated that individuals preferred more to interact with a robot that had the same personality with theirs and that an adapted mixed robot's behavior (gesture and speech) was more engaging and effective than a speech only robot's behavior. Our experiments were done with Nao robot.

[1]  Ran R. Hassin,et al.  Facing faces: studies on the cognitive aspects of physiognomy. , 2000, Journal of personality and social psychology.

[2]  C. Nass,et al.  Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. , 2001, Journal of experimental psychology. Applied.

[3]  Radoslaw Niewiadomski,et al.  Modeling Emotional Expressions as Sequences of Behaviors , 2009, IVA.

[4]  D. McNeill.  Language and Gesture: Gesture in action , 2000 .

[5]  Patrick Pantel,et al.  VerbOcean: Mining the Web for Fine-Grained Semantic Verb Relations , 2004, EMNLP.

[6]  A. Tellegen,et al.  PERSONALITY PROCESSES AND INDIVIDUAL DIFFERENCES An Alternative "Description of Personality": The Big-Five Factor Structure , 2022 .

[7]  Pengcheng Luo,et al.  Synchronized gesture and speech production for humanoid robots , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[8]  Alastair J. Gill,et al.  Taking Care of the Linguistic Features of Extraversion , 2019, Proceedings of the Twenty-Fourth Annual Conference of the Cognitive Science Society.

[9]  L. Morris.  Extraversion and introversion : an interactional perspective , 1979 .

[10]  Diane J. Litman,et al.  Responding to Student Uncertainty During Computer Tutoring: An Experimental Evaluation , 2008, Intelligent Tutoring Systems.

[11]  Michael Neff,et al.  An Examination of Whether People Prefer Agents Whose Gestures Mimic Their Own , 2013, IVA.

[12]  Bilge Mutlu,et al.  Learning-Based Modeling of Multimodal Behaviors for Humanlike Robots , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[13]  Max Coltheart,et al.  The MRC Psycholinguistic Database , 1981 .

[14]  H. Eysenck.  Dimensions of personality : 16, 5 or 3? ― Criteria for a taxonomic paradigm , 1991 .

[15]  Jaap J. A. Denissen,et al.  Emerging late adolescent friendship networks and Big Five personality traits: a social network approach. , 2010, Journal of personality.

[16]  I. René J. A. te Boekhorst,et al.  Is this robot like me? Links between human and robot personality traits , 2005, 5th IEEE-RAS International Conference on Humanoid Robots, 2005..

[17]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[18]  I. Deary,et al.  Personality traits, 2nd ed. , 2003 .

[19]  Maurizio Mancini,et al.  Formational parameters and adaptive prototype instantiation for MPEG-4 compliant gesture synthesis , 2002, Proceedings of Computer Animation 2002 (CA 2002).

[20]  Sarah N. Woods,et al.  Are robots like people?: Relationships between participant and robot personality traits in human–robot interaction studies , 2007 .

[21]  Angel P. del Pobil,et al.  The Law of Attraction in Human-Robot Interaction: , 2012 .

[22]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[23]  H. Giles,et al.  Speech style and social evaluation , 1975 .

[24]  J. Bargh,et al.  Automaticity of social behavior: direct effects of trait construct and stereotype-activation on action. , 1996, Journal of personality and social psychology.

[25]  Adriana Tapus,et al.  Socially Assistive Robots: The Link between Personality, Empathy, Physiological Signals, and Task Performance , 2008, AAAI Spring Symposium: Emotion, Personality, and Social Behavior.

[26]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[27]  C. Nass,et al.  How “Real” Are Computer Personalities? , 1996 .

[28]  M.J. Mataric,et al.  Hands-off assistive robotics for post-stroke arm rehabilitation , 2005, 9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005..

[29]  Henrik I. Christensen,et al.  Embodied social interaction for robots , 2005 .

[30]  S. Gosling,et al.  Personality in its natural habitat: manifestations and implicit folk theories of personality in daily life. , 2006, Journal of personality and social psychology.

[31]  Dennis Windhouwer.  The effects of the task context on the perceived personality of a Nao robot , 2012 .

[32]  Adriana Tapus,et al.  Speech to Head Gesture Mapping in Multimodal Human-Robot Interaction , 2011, ECMR.

[33]  Kevin Leuwerink.  A Robot With Personality: Interacting With a Group of Humans , 2012 .

[34]  John L. Williams.  Personal space and its relation to extraversion-introversion. , 1971 .

[35]  Barbara J. Grosz,et al.  TEAM: A Transportable Natural-Language Interface System , 1983, ANLP.

[36]  Silvia Rossi,et al.  A dialogue system for multimodal human-robot interaction , 2013, ICMI '13.

[37]  J. Cappella,et al.  Talk and Silence Sequences in Informal Conversations III: Interspeaker Influence , 1981 .

[38]  Justine Cassell,et al.  Human conversation as a system framework: designing embodied conversational agents , 2001 .

[39]  Stephen J. Roberts,et al.  Coupled hidden Markov models for biosignal interaction modelling , 2000 .

[40]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[41]  K. Scherer,et al.  Social Markers in Speech , 1980 .

[42]  Marilyn A. Walker,et al.  Controlling User Perceptions of Linguistic Style: Trainable Generation of Personality Traits , 2011, CL.

[43]  Adriana Tapus,et al.  Prosody-driven robot arm gestures generation in human-robot interaction , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[44]  Timothy Leary,et al.  Interpersonal diagnosis of personality , 1957 .

[45]  H. S. Sullivan.  The interpersonal theory of psychiatry , 1953 .

[46]  J. Pennebaker,et al.  Linguistic styles: language use as an individual difference. , 1999, Journal of personality and social psychology.

[47]  Ehud Reiter,et al.  Book Reviews: Building Natural Language Generation Systems , 2000, CL.

[48]  E. Hall,et al.  The Hidden Dimension , 1970 .

[49]  Amir Aly,et al.  Towards an Interactive Human-Robot Relationship: Developing a Customized Robot Behavior to Human Profile. (Vers une relation Homme-Robot Interactive : développement d'un comportement du Robot adapté au Profil de l'Homme) , 2014 .

[50]  Adriana Tapus,et al.  Prosody-based adaptive metaphoric head and arm gestures synthesis in human robot interaction , 2013, 2013 16th International Conference on Advanced Robotics (ICAR).

[51]  Marilyn A. Walker,et al.  Trainable Sentence Planning for Complex Information Presentations in Spoken Dialog Systems , 2004, ACL.

[52]  G. Beattie,et al.  Do metaphoric gestures influence how a message is perceived? The effects of metaphoric gesture-speech matches and mismatches on semantic communication and social judgment , 2012 .

[53]  Stefan Kopp,et al.  Towards a Common Framework for Multimodal Generation: The Behavior Markup Language , 2006, IVA.

[54]  Catherine Pelachaud,et al.  Generating Co-speech Gestures for the Humanoid Robot NAO through BML , 2011, Gesture Workshop.

[55]  Ingrid Zukerman,et al.  Natural Language Processing and User Modeling: Synergies and Limitations , 2001, User Modeling and User-Adapted Interaction.

[56]  Hiroshi Nakajima,et al.  The functionality of human-machine collaboration systems - mind model and social behavior , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[57]  Myers,et al.  Gifts Differing: Understanding Personality Type , 1980 .

[58]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[59]  J. Pittam.  Voice in Social Interaction: An Interdisciplinary Approach , 1994 .

[60]  Benoit Lavoie,et al.  A Fast and Portable Realizer for Text Generation Systems , 1997, ANLP.

[61]  A. Buss,et al.  Personality Traits , 1973 .

[62]  A. Furnham,et al.  Extraversion: The Unloved Variable in Applied Linguistic Research , 1999 .

[63]  D. Byrne,et al.  Similarity and awareness of similarity of personality characteristics as determinants of attraction. , 1969 .

[64]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[65]  Marilyn A. Walker,et al.  Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text , 2007, J. Artif. Intell. Res..

[66]  R. Baaren,et al.  Mimicry for money: Behavioral consequences of imitation , 2003 .

[67]  Thomas Rist,et al.  The automated design of believable dialogues for animated presentation teams , 2001 .

[68]  Dick P. H. Barelds,et al.  Do People Know What They Want: A Similar or Complementary Partner? , 2008 .

[69]  鈴木 聡.  Media Equation 研究の背景と動向 , 2011 .

[70]  R. Riggio,et al.  Impression formation: the role of expressive behavior. , 1986, Journal of personality and social psychology.

[71]  H. Eysenck.  The structure of human personality , 1953 .

[72]  R. Lippa,et al.  The Relation of Gender, Personality, and Intelligence to Judges' Accuracy in Judging Strangers' Personality From Brief Video Segments , 2000 .

[73]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[74]  C. Nass,et al.  Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. , 2001 .

[75]  Adriana Tapus,et al.  An Integrated Model of Speech to Arm Gestures Mapping in Human-Robot Interaction , 2012 .

[76]  A. Furnham.  Language and personality. , 1990 .

[77]  K. M. Lee,et al.  Can robots manifest personality? : An empirical test of personality recognition, social responses, and social presence in human-robot interaction , 2006 .

[78]  K. Vogel,et al.  L'interlangue et la personnalité de l'apprenant , 1986 .

[79]  Maurizio Mancini,et al.  Distinctiveness in multimodal behaviors , 2008, AAMAS.

[80]  Richard E. Maurer,et al.  Effect of Postural Congruence on Client's Perception of Counselor Empathy. , 1983 .

[81]  Timothy W. Finin,et al.  Natural language interactions with artificial experts , 1986, Proceedings of the IEEE.

[82]  Frank J. Bernieri,et al.  Coordinated movement and rapport in teacher-student interactions , 1988 .

[83]  Robert Dale,et al.  Building Natural Language Generation Systems: Figures , 2000 .

[84]  Justine Cassell,et al.  Negotiated Collusion: Modeling Social Language and its Relationship Effects in Intelligent Agents , 2003, User Modeling and User-Adapted Interaction.

[85]  M. Mancini,et al.  Speaking with Emotions , 2004 .

[86]  J. B. Murray.  Review of Research on the Myers-Briggs Type Indicator , 1990 .

[87]  T. Chartrand,et al.  The chameleon effect: the perception-behavior link and social interaction. , 1999, Journal of personality and social psychology.

[88]  J. Bailenson,et al.  Digital Chameleons , 2005, Psychological science.

[89]  Adriana Tapus,et al.  User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy , 2008, Intell. Serv. Robotics.

[90]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents: Research Articles , 2004 .

[91]  Alfred Kobsa,et al.  User Models in Dialog Systems , 1989, Symbolic Computation.

[92]  Ingrid Zukerman,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Predictive Statistical Models for User Modeling , 1999 .

[93]  Diane J. Litman,et al.  Investigating Human Tutor Responses to Student Uncertainty for Adaptive System Development , 2007, ACII.

[94]  S. Roberts,et al.  Estimation of coupled hidden Markov models with application to biosignal interaction modelling , 2000, Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.00TH8501).

[95]  D. McNeill.  Gesture and Thought , 2005 .

[96]  Clifford Nass,et al.  Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics , 2000, Int. J. Hum. Comput. Stud..

[97]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[98]  C. G. Jung.  Psychological Types , 2000 .