Multi-party, multi-role comprehensive listening behavior

Realizing effective listening behavior in virtual humans has become a key area of research, especially as research has sought to realize more complex social scenarios involving multiple participants and bystanders. A human listener’s nonverbal behavior is conditioned by a variety of factors, from current speaker’s behavior to the listener’s role and desire to participate in the conversation and unfolding comprehension of the speaker. Similarly, we seek to create virtual humans able to provide feedback based on their participatory goals and their unfolding understanding of, and reaction to, the relevance of what the speaker is saying as the speaker speaks. Based on a survey of existing psychological literature as well as recent technological advances in recognition and partial understanding of natural language, we describe a model of how to integrate these factors into a virtual human that behaves consistently with these goals. We then discuss how the model is implemented into a virtual human architecture and present an evaluation of behaviors used in the model.

[1]  Stacy Marsella,et al.  Natural Behavior of a Listening Agent , 2005, IVA.

[2]  Matthew P. Aylett,et al.  Intelligent Virtual Agents , 2010, Lecture Notes in Computer Science.

[3]  C. Goodwin.  Conversational Organization: Interaction Between Speakers and Hearers , 1981 .

[4]  Dirk Heylen,et al.  Backchannel Strategies for Artificial Listeners , 2010, IVA.

[5]  J. Bavelas,et al.  Listeners as co-narrators. , 2000, Journal of personality and social psychology.

[6]  D. Cicchetti.  Emotion and Adaptation , 1993 .

[7]  R. Lazarus.  Emotion and Adaptation , 1991 .

[8]  David R. Traum,et al.  Dynamic movement and positioning of embodied agents in multiparty conversations , 2007, AAMAS '07.

[9]  Roel Vertegaal,et al.  Effects of Gaze on Multiparty Mediated Communication , 2000, Graphics Interface.

[10]  Stefan Kopp,et al.  Towards a Common Framework for Multimodal Generation: The Behavior Markup Language , 2006, IVA.

[11]  V. Yngve.  On getting a word in edgewise , 1970 .

[12]  R. Riggio,et al.  Effect of individual differences in nonverbal expressiveness on transmission of emotion , 1981 .

[13]  Elisabetta Bevacqua,et al.  Multimodal Backchannels for Embodied Conversational Agents , 2010, IVA.

[14]  Stacy Marsella,et al.  Predicting Speaker Head Nods and the Effects of Affective Information , 2010, IEEE Transactions on Multimedia.

[15]  E. Goffman,et al.  Forms of talk , 1982 .

[16]  M. Argyle,et al.  The Effects of Visibility on Interaction in a Dyad , 1968 .

[17]  Ning Wang,et al.  Creating Rapport with Virtual Agents , 2007, IVA.

[18]  Stefan Kopp,et al.  The Next Step towards a Function Markup Language , 2008, IVA.

[19]  W. Hanks.  Language & communicative practices , 1995 .

[20]  Kristinn R. Thórisson,et al.  Fluid Semantic Back-Channel Feedback in Dialogue: Challenges and Progress , 2007, IVA.

[21]  David R. Traum,et al.  Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents , 2008, IVA.

[22]  Adam Kendon,et al.  Some uses of the head shake , 2002 .

[23]  Evelyn Z. McClave.  Linguistic functions of head movements in the context of speech , 2000 .

[24]  David DeVault,et al.  Incremental interpretation and prediction of utterance meaning for interactive dialogue , 2011, Dialogue Discourse.

[25]  Howard S. Friedman,et al.  Some Effects of Gaze on Subjects Motivated to Seek or to Avoid Social Comparison. , 1978 .

[26]  A. Dittmann,et al.  Relationship between vocalizations and head nods as listener responses. , 1968, Journal of personality and social psychology.

[27]  Stacy Marsella,et al.  EMA: A process model of appraisal dynamics , 2009, Cognitive Systems Research.

[28]  A. Kendon.  Some Relationships Between Body Motion and Speech , 1972 .

[29]  Louis-Philippe Morency,et al.  A probabilistic multimodal approach for predicting listener backchannels , 2009, Autonomous Agents and Multi-Agent Systems.

[30]  K. Ikeda.  Triadic exchange pattern in multiparty communication: a case study of conversational narrative among friends , 2009 .

[31]  David DeVault,et al.  Incremental Dialogue Understanding and Feedback for Multiparty, Multimodal Conversation , 2012, IVA.

[32]  D. Heylen.  Challenges ahead: head movements and other social acts during conversations , 2005 .

[33]  Hilary M.W. Callan,et al.  Attention and advertence in human groups , 1973 .

[34]  Norman I. Badler,et al.  Visual Attention and Eye Gaze During Multiparty Conversations with Distractions , 2006, IVA.

[35]  Kristinn R. Thórisson,et al.  Fluid Semantic BackChannel Feedback in Dialogue : Challenges and Progress , .

[36]  M. Argyle,et al.  Gaze and Mutual Gaze , 1994, British Journal of Psychiatry.

[37]  Stefan Kopp,et al.  Modeling Embodied Feedback with Virtual Humans , 2006, ZiF Workshop.

[38]  Stacy Marsella,et al.  SmartBody: behavior realization for embodied conversational agents , 2008, AAMAS.

[39]  Dirk Heylen,et al.  Appropriate and Inappropriate Timing of Listener Responses from Multiple Perspectives , 2011, IVA.

[40]  Stacy Marsella,et al.  Nonverbal Behavior Generator for Embodied Conversational Agents , 2006, IVA.

[41]  L. J. Brunner,et al.  Smiles can be back channels. , 1979 .

[42]  A. Kendon.  Conducting Interaction: Patterns of Behavior in Focused Encounters , 1990 .

[43]  et al.,et al.  At the Virtual Frontier: Introducing Gunslinger, a Multi-Character, Mixed-Reality, Story-Driven Experience , 2009, IVA.

[44]  Greg Urban.  Language and Communicative Practices , 1998 .