Generating finely synchronized gesture and speech for humanoid robots: A closed-loop approach

Previous work on the production of communicative robot gesture has not sufficiently addressed the challenge of speech-gesture synchrony. We propose a novel closed-loop multimodal scheduler that comprises two features to improve the synchronization process. First, the scheduler integrates an experimentally fitted forward model at the behavior planning stage to provide a more accurate estimation of the robot's gesture preparation time. Second, the model incorporates a feedback-based adaptation mechanism which allows for on-line adjustment of the synchronization of the two modalities during execution.

[1]  Pengcheng Luo,et al.  Synchronized gesture and speech production for humanoid robots , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[2]  Stefan Kopp,et al.  Synthesizing multimodal utterances for conversational agents , 2004, Comput. Animat. Virtual Worlds.

[3]  Jan Peter De Ruiter,et al.  On the audiovisual integration of speech and gesture , 2012 .

[4]  Catherine Pelachaud,et al.  Design and implementation of an expressive gesture model for a humanoid robot , 2011, 2011 11th IEEE-RAS International Conference on Humanoid Robots.

[5]  Zeshu Shao,et al.  The Role of Synchrony and Ambiguity in Speech–Gesture Integration during Comprehension , 2011, Journal of Cognitive Neuroscience.

[6]  Autumn B. Hostetter,et al.  When do gestures communicate? A meta-analysis. , 2011, Psychological bulletin.

[7]  Marc Schröder,et al.  The German Text-to-Speech Synthesis System MARY: A Tool for Research, Development and Teaching , 2003, Int. J. Speech Technol..

[8]  D. McNeill.  Gesture and Thought , 2005 .

[9]  Justine Cassell,et al.  Human conversation as a system framework: designing embodied conversational agents , 2001 .

[10]  Stefan Kopp,et al.  Generation and Evaluation of Communicative Robot Gesture , 2012, Int. J. Soc. Robotics.

[11]  Scott T. Grafton,et al.  Forward modeling allows feedback control for fast reaching movements , 2000, Trends in Cognitive Sciences.

[12]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[13]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[14]  Maja J. Mataric,et al.  An architecture for rehabilitation task practice in socially assistive human-robot interaction , 2010, 19th International Symposium in Robot and Human Interactive Communication.

[15]  Matthew Stone,et al.  Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems , 1999 .

[16]  Radoslaw Niewiadomski,et al.  Greta: an interactive expressive ECA system , 2009, AAMAS.

[17]  Michael Gienger,et al.  Task-oriented whole body motion for humanoid robots , 2005, 5th IEEE-RAS International Conference on Humanoid Robots, 2005..

[18]  Stefan Kopp,et al.  MURML: A Multimodal Utterance Representation Markup Language for Conversational Agents , 2002 .

[19]  Mark Dunn,et al.  Predictive Behavior Generation - A Sensor-Based Walking and Reaching Architecture for Humanoid Robots , 2007, AMS.

[20]  Christian Goerick,et al.  Instant prediction for reactive motions with planning , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[21]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[22]  Sriram Subramanian,et al.  Conversational gestures in human-robot interaction , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.