Multimodal Sensing, Interpretation and Copying of Movements by a Virtual Agent

We present a scenario whereby an agent senses, interprets and copies a range of facial and gesture expression from a person in the real-world. Input is obtained via a video camera and processed initially using computer vision techniques. It is then processed further in a framework for agent perception, planning and behaviour generation in order to perceive, interpret and copy a number of gestures and facial expressions corresponding to those made by the human. By perceive, we mean that the copied behaviour may not be an exact duplicate of the behaviour made by the human and sensed by the agent, but may rather be based on some level of interpretation of the behaviour. Thus, the copied behaviour may be altered and need not share all of the characteristics of the original made by the human.

[1]  Takeo Kanade,et al.  Recognizing lower face action units for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[2]  Norman I. Badler,et al.  FacEMOTE: qualitative parametric modifiers for facial animations , 2002, SCA '02.

[3]  P. Gallaher.  Individual differences in nonverbal behavior : dimensions of style , 1992 .

[4]  A. Murat Tekalp,et al.  Face and 2-D mesh animation in MPEG-4 , 2000, Signal Process. Image Commun..

[5]  Takeo Kanade,et al.  Facial Expression Analysis , 2011, AMFG.

[6]  Antonio Camurri,et al.  Gesture-Based Communication in Human-Computer Interaction , 2003, Lecture Notes in Computer Science.

[7]  Alan Wexelblat,et al.  An approach to natural gesture in virtual environments , 1995, TCHI.

[8]  Catherine Pelachaud,et al.  Computational Model of Believable Conversational Agents , 2003, Communication in Multiagent Systems.

[9]  Maurizio Mancini,et al.  Formational parameters and adaptive prototype instantiation for MPEG-4 compliant gesture synthesis , 2002, Proceedings of Computer Animation 2002 (CA 2002).

[10]  Kostas Karpouzis,et al.  Parameterized Facial Expression Synthesis Based on MPEG-4 , 2002, EURASIP J. Adv. Signal Process..

[11]  K. Scherer,et al.  Cues and channels in emotion recognition. , 1986 .

[12]  P. Ekman,et al.  Approaches To Emotion , 1985 .

[13]  Maurizio Mancini,et al.  Levels of Representation in the Annotation of Emotion for the Specification of Expressivity in ECAs , 2005, IVA.

[14]  Norman I. Badler,et al.  The EMOTE model for effort and shape , 2000, SIGGRAPH.

[15]  Stacy Marsella,et al.  Natural Behavior of a Listening Agent , 2005, IVA.

[16]  Maurizio Mancini,et al.  Implementing Expressive Gesture Synthesis for Embodied Conversational Agents , 2005, Gesture Workshop.

[17]  J. Harrigan,et al.  LISTENERS' BODY MOVEMENTS AND SPEAKING TURNS , 1985 .

[18]  Frank E. Pollick,et al.  The Features People Use to Recognize Human Movement Style , 2003, Gesture Workshop.

[19]  J. Breese,et al.  Emotion and personality in a conversational agent , 2001 .

[20]  Stefan Kopp,et al.  Imitation Games with an Artificial Agent: From Mimicking to Understanding Shape-Related Iconic Gestures , 2003, Gesture Workshop.

[21]  Gwen Littlewort,et al.  Recognizing facial expression: machine learning and application to spontaneous behavior , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Maurizio Mancini,et al.  Design and evaluation of expressive gesture synthesis for embodied conversational agents , 2005, AAMAS '05.

[23]  Marc-Philippe Huget,et al.  Communication in Multiagent Systems , 2003 .

[24]  G. Johansson.  Visual perception of biological motion and a model for its analysis , 1973 .

[25]  Christopher E. Peters.  Direction of Attention Perception for Conversation Initiation in Virtual Environments , 2005, IVA.

[26]  K. Rapantzikos,et al.  AN ENHANCED SPATIOTEMPORAL VISUAL ATTENTION MODEL FOR SPORTS VIDEO ANALYSIS , 2005 .

[27]  Trevor Darrell,et al.  Contextual recognition of head gestures , 2005, ICMI '05.

[28]  Marian Stewart Bartlett,et al.  Classifying Facial Actions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  P. Gallaher.  Individual differences in nonverbal behavior: Dimensions of style. , 1992 .

[30]  Ying Wu,et al.  Hand modeling, analysis and recognition , 2001, IEEE Signal Process. Mag..