Facilitating multiparty dialog with gaze, gesture, and speech

We study how synchronized gaze, gesture and speech rendered by an embodied conversational agent can influence the flow of conversations in multiparty settings. We begin by reviewing a computational framework for turn-taking that provides the foundation for tracking and communicating intentions to hold, release, or take control of the conversational floor. We then present implementation aspects of this model in an embodied conversational agent. Empirical results with this model in a shared task setting indicate that the various verbal and non-verbal cues used by the avatar can effectively shape the multiparty conversational dynamics. In addition, we identify and discuss several context variables which impact the turn allocation process.

[1]  Eric Horvitz,et al.  Models for Multiparty Engagement in Open-World Dialog , 2009, SIGDIAL Conference.

[2]  Kristinn R. Thórisson,et al.  Natural Turn-Taking Needs No Manual: Computational Theory and Model, from Perception to Action , 2002 .

[3]  S. Duncan,et al.  Some Signals and Rules for Taking Speaking Turns in Conversations , 1972 .

[4]  C. Goodwin.  Restarts, Pauses, and the Achievement of a State of Mutual Gaze at Turn‐Beginning , 1980 .

[5]  David R. Traum,et al.  Embodied agents for multi-party dialogue in immersive virtual worlds , 2002, AAMAS '02.

[6]  Laurent Itti,et al.  Realistic avatar eye and head animation using a neurobiological model of visual attention , 2004, SPIE Optics + Photonics.

[7]  Eric Horvitz,et al.  Computational Models for Multiparty Turn-Taking , 2010 .

[8]  E. Schegloff.  Overlapping talk and the organization of turn-taking for conversation , 2000, Language in Society.

[9]  Maxine Eskénazi,et al.  A Finite-State Turn-Taking Model for Spoken Dialog Systems , 2009, NAACL.

[10]  B. Granström,et al.  NATURAL TURN-TAKING NEEDS NO MANUAL : COMPUTATIONAL THEORY AND MODEL , FROM PERCEPTION TO ACTION , 2002 .

[11]  Eric Horvitz,et al.  Dialog in the open world: platform and applications , 2009, ICMI-MLMI '09.

[12]  M. Knapp,et al.  Turn-Taking in Conversations. , 1975 .

[13]  Gérard Bailly,et al.  Scrutinizing Natural Scenes: Controlling the Gaze of an Embodied Conversational Agent , 2007, IVA.

[14]  E. Schegloff,et al.  A simplest systematics for the organization of turn-taking for conversation , 1974 .

[15]  J. Cassell,et al.  Turn taking vs. Discourse Structure: How Best to Model Multimodal Conversation , 1998 .

[16]  Sven Behnke,et al.  Integrating vision and speech for conversations with multiple persons , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Emanuel A. Schegloff,et al.  Accounts of Conduct in Interaction: Interruption, Overlap, and Turn-Taking , 2001 .

[18]  Takayuki Kanda,et al.  Footing in human-robot conversations: How robots might shape participant roles using gaze cues , 2009, 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[19]  H. H. Clark,et al.  Hearers and speech acts , 1982 .