On the simulation of interactive non-verbal behaviour in virtual humans

Development of virtual humans has focused mainly in two broad areas - conversational agents and computer game characters. Computer game characters have traditionally been action-oriented - focused on the game-play - and conversational agents have been focused on sensible/intelligent conversation. While virtual humans have incorporated some form of non-verbal behaviour, this has been quite limited and more importantly not connected or connected very loosely with the behaviour of a real human interacting with the virtual human - due to a lack of sensor data and no system to respond to that data. The interactional aspect of non-verbal behaviour is highly important in human-human interactions and previous research has demonstrated that people treat media (and therefore virtual humans) as real people, and so interactive non-verbal behaviour is also important in the development of virtual humans. This paper presents the challenges in creating virtual humans that are non-verbally interactive and drawing corollaries with the development history of control systems in robotics presents some approaches to solving these challenges - specifically using behaviour based systems - and shows how an order of magnitude increase in response time of virtual humans in conversation can be obtained and that the development of rapidly responding non-verbal behaviours can start with just a few behaviours with more behaviours added without difficulty later in development.

[1]  M. Minsky.  The Society of Mind , 1986 .

[2]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[3]  W. Walter.  The Living Brain , 1963 .

[4]  E. Gat.  On Three-Layer Architectures , 1997 .

[5]  Stefan Kopp,et al.  Imitation Games with an Artificial Agent: From Mimicking to Understanding Shape-Related Iconic Gestures , 2003, Gesture Workshop.

[6]  Rodney A. Brooks.  PLANNING IS JUST A WAY OF AVOIDING FIGURING OUT WHAT TO DO NEXT , 1987 .

[7]  Judy M. Vance,et al.  From gesture recognition to functional motion analysis: quantitative techniques for the application and evaluation of human motion , 2005 .

[8]  Louis-Philippe Morency,et al.  The effect of head-nod recognition in human-robot conversation , 2006, HRI '06.

[9]  Ronald C. Arkin,et al.  An Behavior-based Robotics , 1998 .

[10]  Hao Yan,et al.  Paired speech and gesture generation in embodied conversational agents , 2000 .

[11]  D. McNeill.  Gesture and Thought , 2005 .

[12]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[13]  Oliver G. Selfridge,et al.  Pattern recognition by machine , 1960 .

[14]  V. Braitenberg.  Vehicles, Experiments in Synthetic Psychology , 1984 .

[15]  Joseph Weizenbaum,et al.  and Machine , 1977 .

[16]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[17]  Victor R. Lesser,et al.  The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty , 1980, CSUR.

[18]  Ken Perlin,et al.  Real Time Responsive Animation with Personality , 1995, IEEE Trans. Vis. Comput. Graph..

[19]  J. P. Foley,et al.  Gesture and Environment , 1942 .

[20]  Susanne van Mulken,et al.  The impact of animated interface agents: a review of empirical research , 2000, Int. J. Hum. Comput. Stud..

[21]  Stefan Kopp,et al.  Content in Context: Generating Language and Iconic Gestures without a Gestionary , 2004 .

[22]  Norbert Wiener,et al.  Cybernetics: Control and Communication in the Animal and the Machine. , 1949 .

[23]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[24]  Hatice Gunes,et al.  Face and Body Gesture Recognition for a Vision-Based Multimodal Analyzer , 2004, VIP.

[25]  Marcus Redley.  Face-to-face interaction in research interviews , 1996 .

[26]  Francis K. H. Quek,et al.  Gestural Hand Motion Oscillation and Symmetries for Multimodal Discourse: Detection and Analysis , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[27]  G. Miller,et al.  Handbook of Interpersonal Communication , 1985 .

[28]  Radoslaw Niewiadomski,et al.  Multimodal Complex Emotions: Gesture Expressivity and Blended Facial Expressions , 2006, Int. J. Humanoid Robotics.