Describing and generating multimodal contents featuring affective lifelike agents with MPML

In this paper, we provide an overview of our research on multimodal media and contents using embodied lifelike agents. In particular we describe our research centered on MPML (Multimodal Presentation Markup Language). MPML allows people to write and produce multimodal contents easily, and serves as a core for integrating various components and functionalities important for multimodal media. To demonstrate the benefits and usability of MPML in a variety of environments including animated Web, 3D VRML space, mobile phones, and the physical world with a humanoid robot, several versions of MPML have been developed while keeping its basic format. Since emotional behavior of the agent is an important factor for making agents lifelike and for being accepted by people as an attractive and friendly human-computer interaction style, emotion-related functions have been emphasized in MPML. In order to alleviate the workload of authoring the contents, it is also required to endow the agents with a certain level of autonomy. We show some of our approaches towards this end.

[1]  John L. Arnott,et al.  Implementation and testing of a system for producing emotion-by-rule in synthetic speech , 1995, Speech Commun..

[2]  Mitsuru Ishizuka,et al.  THE EMPATHIC COMPANION: A CHARACTER-BASED INTERFACE THAT ADDRESSES USERS' AFFECTIVE STATES , 2005, Appl. Artif. Intell..

[3]  Justine Cassell,et al.  BEAT: the Behavior Expression Animation Toolkit , 2001, Life-like characters.

[4]  Zhisheng Huang,et al.  STEP: a Scripting Language for Embodied Agents , 2004, Life-like characters.

[5]  Mitsuru Ishizuka,et al.  Empathic Embodied Interfaces: Addressing Users' Affective State: Embodied Interfaces That Address Users' Physiological State , 2004, ADS.

[6]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[7]  Mitsuru Ishizuka,et al.  Evaluating Affective Feedback of the 3D Agent Max in a Competitive Cards Game , 2005, ACII.

[8]  Mitsuru Ishizuka,et al.  Evaluation of an Embodied Conversational Agent with Affective Behavior , 2003 .

[9]  R. McCrae,et al.  An introduction to the five-factor model and its applications. , 1992, Journal of personality.

[10]  Naoaki Okazaki,et al.  An extension of the multimodal presentation markup language (MPML) to a three-dimensional VRML space , 2005, Systems and Computers in Japan.

[11]  Mitsuru Ishizuka,et al.  Journal of Visual Languages , 2002 .

[12]  Mitsuru Ishizuka,et al.  Life-like characters - tools, affective functions, and applications , 2004, Life-like characters.

[13]  Toyoaki Nishida,et al.  Converting Text into Agent Animations: Assigning Gestures to Text , 2004, HLT-NAACL.

[14]  Mitsuru Ishizuka,et al.  Humanoid Robot Presentation through Multimodal Presentation Markup Language MPML-HR , 2005 .

[15]  Rosalind W. Picard.  Affective Computing , 1997 .

[16]  Mark Steedman,et al.  APML, a Markup Language for Believable Behavior Generation , 2004, Life-like characters.

[17]  Sylvain Descamps.  A Multimodal Presentation Mark-up Language for Enhanced Affective Presentation , 2001 .

[18]  P. Lang.  The emotion probe. Studies of motivation and attention. , 1995, The American psychologist.

[19]  Brigitte Krenn,et al.  RRL: A Rich Representation Language for the Description of Agent Behaviour in NECA , 2004, ArXiv.

[20]  Mitsuru Ishizuka,et al.  MPML 2.0e : Multimodal Presentation Markup Language supporting Emotion Expression , 2000 .

[21]  Mitsuru Ishizuka,et al.  Social role awareness in animated agents , 2001, AGENTS '01.

[22]  Andrew Marriott,et al.  VHML – Uncertainties and Problems . A discussion ... , 2002 .

[23]  Naoaki Okazaki,et al.  Improving Chronological Sentence Ordering by Precedence Relation , 2004, COLING.

[24]  J. Breese,et al.  Emotion and personality in a conversational agent , 2001 .

[25]  Justine Cassell,et al.  Embodied conversational interface agents , 2000, CACM.

[26]  石塚 満,et al.  K-40 Enhancing Conversational Flexibility in Multimodal Interactions with Embodied Lifelike Agents , 2002 .

[27]  Helmut Prendinger,et al.  Eye movement as an indicator of users'involvement with embodied interfaces at the low level , 2005 .

[28]  Catherine Pelachaud,et al.  From brows to trust: evaluating embodied conversational agents , 2004 .

[29]  Mitsuru Ishizuka,et al.  Human Physiology as a Basis for Designing and Evaluating Affective Communication with Life-Like Characters , 2005, IEICE Trans. Inf. Syst..

[30]  Igor S. Pandžić,et al.  Automatic Content Production for an Autonomous Speaker Agent , 2005 .

[31]  Mitsuru Ishizuka,et al.  A 3D Agent with Synthetic Face and Semiautonomous Behavior for Multimodal Presentations , 2001 .

[32]  Mitsuru Ishizuka,et al.  Persona Effect Revisited , 2003, IVA.

[33]  Mitsuru Ishizuka,et al.  MPML: A Multimodal Presentation Markup Language with Character Agent Control Functions , 2000, WebNet.

[34]  Mitsuru Ishizuka,et al.  Multimodal Presentation Markup Language MPML with emotion expression functions attached , 2000, Proceedings International Symposium on Multimedia Software Engineering.

[35]  James F. Allen,et al.  Toward Conversational Human-Computer Interaction , 2001, AI Mag..

[36]  Michael Kipp,et al.  From Human Gesture to Synthetic Action , 2001 .

[37]  Mitsuru Ishizuka,et al.  Agents That Talk Back (Sometimes): Filter Programs for Affective Communication , 2001 .

[38]  Mitsuru Ishizuka,et al.  Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game , 2005, Int. J. Hum. Comput. Stud..

[39]  H. Iba,et al.  Humanoid robot presentation controlled by multimodal presentation markup language MPML , 2004, RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759).

[40]  Jun Nakazoe,et al.  Modulation-type capacitive electrode driven by small AC current for measuring DC biological signals , 2005 .

[41]  Mitsuru Ishizuka,et al.  Summarization of Dynamic Content in Web Collections , 2004, PKDD.

[42]  Naoaki Okazaki,et al.  Scripting and Evaluating Affective Interactions with Embodied Conversational Agents , 2004, Künstliche Intell..

[43]  Mitsuru Ishizuka,et al.  Scripting affective communication with life-like characters in web-based interaction systems , 2002, Appl. Artif. Intell..

[44]  Barbara Hayes-Roth.  What Makes Characters Seem Life-Like? , 2004, Life-like characters.

[45]  Mitsuru Ishizuka,et al.  Scripting the Bodies and Minds of Life-Like Characters , 2002, PRICAI.

[46]  Mitsuru Ishizuka,et al.  Persona Effect Revisited --- Using Bio-signals to Measure and Reflect the Impact of Character-bas , 2003 .

[47]  Norman I. Badler,et al.  A Parameterized Action Representation for Virtual Human Agents , 1998 .

[48]  Oliviero Stock,et al.  Multimodal intelligent information presentation , 2005 .

[49]  Henry Lieberman,et al.  A model of textual affect sensing using real-world knowledge , 2003, IUI '03.

[50]  C. Fellbaum.  An Electronic Lexical Database , 1998 .

[51]  Naoaki Okazaki,et al.  Markup Language MPML-VR for a 3 D Virtual Space , 2002 .

[52]  James F. Allen,et al.  Towards Conversational Human-Computer Interaction , 2000 .

[53]  Mitsuru Ishizuka,et al.  'Auto-Presentation': a multi-agent system for building automatic multi-modal presentation of a topic from World Wide Web information , 2005, IEEE/WIC/ACM International Conference on Intelligent Agent Technology.

[54]  Mitsuru Ishizuka,et al.  Emotion Estimation and Reasoning Based on Affective Textual Interaction , 2005, ACII.

[55]  Daniel Thalmann,et al.  Two Approaches to Scripting Character Animation , 2002 .

[56]  Mitsuru Ishizuka,et al.  MAKING THE WEB EMOTIONAL: AUTHORING MULTIMODAL PRESENTATIONS USING A SYNTHETIC 3D AGENT , 2001 .

[57]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[58]  Danushka Bollegala,et al.  A Machine Learning Approach to Sentence Ordering for Multidocument Summarization and Its Evaluation , 2005, IJCNLP.

[59]  Mitsuru Ishizuka,et al.  Enhancing conversational flexibility in multimodal interactions with embodied lifelike agent , 2003, IUI '03.

[60]  Mitsuru Ishizuka,et al.  A Scripting Language for Multimodal Presentation on Mobile Phones , 2003, IVA.

[61]  Mitsuru Ishizuka,et al.  Bringing affective behaviour to presentation agents , 2001, Proceedings 21st International Conference on Distributed Computing Systems Workshops.

[62]  Matthew Stone,et al.  Speaking with hands: creating animated conversational characters from recordings of human performance , 2004, SIGGRAPH 2004.

[63]  Carlo Strapparava,et al.  Developing Affective Lexical Resources , 2004, PsychNology J..

[64]  James C. Lester,et al.  The persona effect: affective impact of animated pedagogical agents , 1997, CHI.

[65]  Naoaki Okazaki,et al.  An extension of the multimodal presentation markup language (MPML) to a three-dimensional VRML space , 2005 .

[66]  Mitsuru Ishizuka,et al.  Enhancing Conversational Flexibility in Multimodal Interactions with Embodied Lifelike Agents , 2002 .

[67]  Brigitte Krenn,et al.  Defining the Gesticon : Language and Gesture Coordination for Interacting Embodied Agents , 2004 .

[68]  Mitsuru Ishizuka,et al.  SCREAM: scripting emotion-based agent minds , 2002, AAMAS '02.

[69]  Naoaki Okazaki,et al.  TISS: An Integrated Summarization System for TSC-3 , 2004, NTCIR.

[70]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[71]  Mitsuru Ishizuka,et al.  Let's talk! Socially intelligent agents for language conversation training , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[72]  Mitsuru Ishizuka,et al.  Visual authoring tool for presentation agent based on Multimodal Presentation Markup Language , 2001, Proceedings Fifth International Conference on Information Visualisation.

[73]  Mitsuru Ishizuka,et al.  Life-Like Characters , 2004, Cognitive Technologies.