An Incremental Multimodal Realizer for Behavior Co-Articulation and Coordination

Human conversations are highly dynamic, responsive interactions. To enter into flexible interactions with humans, a conversational agent must be capable of fluent incremental behavior generation. New utterance content must be integrated seamlessly with ongoing behavior, requiring dynamic application of co-articulation. The timing and shape of the agent's behavior must be adapted on-the-fly to the interlocutor, resulting in natural interpersonal coordination. We present AsapRealizer, a BML 1.0 behavior realizer that achieves these capabilities by building upon, and extending, two state of the art existing realizers, as the result of a collaboration between two research groups.

[1]  Igor S. Pandzic,et al.  Multimodal behavior realization for embodied conversational agents , 2011, Multimedia Tools and Applications.

[2]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[3]  Stefan Kopp,et al.  Towards an integrated model of speech and gesture production for multi-modal robot behavior , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[4]  C. Goodwin.  Between and within: Alternative sequential treatments of continuers and assessments , 1986 .

[5]  Dennis Reidsma,et al.  Multimodal plan representation for adaptable BML scheduling , 2011, Autonomous Agents and Multi-Agent Systems.

[6]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents: Research Articles , 2004 .

[7]  Alexis Héloir,et al.  REAL-TIME ANIMATION OF INTERACTIVE AGENTS: SPECIFICATION AND REALIZATION , 2010, Appl. Artif. Intell..

[8]  Yuyu Xu,et al.  Demonstrating and Testing the BML Compliance of BML Realizers , 2011, IVA.

[9]  Bernard Rimé,et al.  Fundamentals of nonverbal behavior , 1991 .

[10]  Anton Nijholt,et al.  Leading and following with a virtual trainer , 2011 .

[11]  Gabriel Skantze,et al.  A General, Abstract Model of Incremental Dialogue Processing , 2011 .

[12]  Rudolf Arnheim,et al.  Hand and Mind: What Gestures Reveal About Thought by David McNeill (review) , 2017 .

[13]  Dennis Reidsma,et al.  Elckerlyc - A BML Realizer for continuous, multimodal interaction with a Virtual Human , 2009 .

[14]  Herwin van Welbergen,et al.  Real-Time Animation Using a Mix of Physical Simulation and Kinematics , 2009, J. Graphics, GPU, & Game Tools.

[15]  Frank J. Bernieri,et al.  Interpersonal coordination: Behavior matching and interactional synchrony. , 1991 .

[16]  Cláudio T. Silva,et al.  Accelerating Unstructured Volume Rendering with Joint Bilateral Upsampling , 2009, J. Graphics, GPU, & Game Tools.

[17]  Anton Nijholt,et al.  Mutually Coordinated Anticipatory Multimodal Interaction , 2008, COST 2102 Workshop.

[18]  Stefan Kopp,et al.  Towards a Common Framework for Multimodal Generation: The Behavior Markup Language , 2006, IVA.

[19]  Nikolaos G. Bourbakis,et al.  Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction, COST Action 2102 International Conference, Patras, Greece, October 29-31, 2007. Revised Papers , 2008, COST 2102 Workshop.

[20]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[21]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[22]  Dennis Reidsma,et al.  Continuous interaction with a virtual human , 2011, Journal on Multimodal User Interfaces.

[23]  Stacy Marsella,et al.  SmartBody: behavior realization for embodied conversational agents , 2008, AAMAS.