Embodied Conversational Agents: Representation and Intelligence in User Interfaces

How do we decide how to represent an intelligent system in its interface, and how do we decide how the interface represents information about the world and about its own workings to a user? This article addresses these questions by examining the interaction between representation and intelligence in user interfaces. The rubric representation covers at least three topics in this context: (1) how a computational system is represented in its user interface, (2) how the interface conveys its representations of information and the world to human users, and (3) how the system's internal representation affects the human user's interaction with the system. I argue that each of these kinds of representation (of the system, information and the world, the interaction) is key to how users make the kind of attributions of intelligence that facilitate their interactions with intelligent systems. In this vein, it makes sense to represent a systmem as a human in those cases where social collaborative behavior is key and for the system to represent its knowledge to humans in multiple ways on multiple modalities. I demonstrate these claims by discussing issues of representation and intelligence in an embodied conversational agent -- an interface in which the system is represented as a person, information is conveyed to human users by multiple modalities such as voice and hand gestures, and the internal representation is modality independent and both propositional and nonpropositional.

[1]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[2]  Hao Yan,et al.  Coordination and context-dependence in the generation of embodied conversation , 2000, INLG.

[3]  Wolfgang Wahlster,et al.  Readings in Intelligent User Interfaces , 1998 .

[4]  Hao Yan,et al.  Shared reality: physical collaboration with a virtual peer , 2000, CHI Extended Abstracts.

[5]  D. Olson,et al.  Developing theories of mind , 1988 .

[6]  Susan Brennan,et al.  Interaction and feedback in a spoken language system: a theoretical framework , 1995, Knowl. Based Syst..

[7]  Wolfgang Wahlster,et al.  Designing Illustrated Texts: How Language Production Is Influenced by Graphics Generation , 1991, EACL.

[8]  L A Thompson,et al.  Evaluation and integration of speech and pointing gestures during referential understanding. , 1986, Journal of experimental child psychology.

[9]  Hao Yan,et al.  Paired speech and gesture generation in embodied conversational agents , 2000 .

[10]  W. Lewis Johnson,et al.  Animated Agents for Procedural Training in Virtual Reality: Perception, Cognition, and Motor Control , 1999, Appl. Artif. Intell..

[11]  James C. Lester,et al.  Increasing believability in animated pedagogical agents , 1997, AGENTS '97.

[12]  P. L. Adams.  THE ORIGINS OF INTELLIGENCE IN CHILDREN , 1976 .

[13]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[14]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[15]  Jennifer Smith,et al.  GrandChair : conversational collection of grandparents' stories , 2000 .

[16]  Hao Yan.  Paired Speech and Gesture Generation in Embodied Conversational Agents , 2000 .

[17]  S. Duncan,et al.  On the structure of speaker–auditor interaction during speaking turns , 1974, Language in Society.

[18]  Steven K. Feiner,et al.  Automating the generation of coordinated multimedia explanations , 1991, Computer.

[19]  Ben Shneiderman,et al.  Anthropomorphism: from Eliza to Terminator 2 , 1992, CHI.

[20]  Leslie Felperin,et al.  Harry Potter and the Chamber of Secrets (Book) , 2003 .

[21]  M. Alibali,et al.  Transitions in concept acquisition: using the hand to read the mind. , 1993, Psychological review.

[22]  Christopher R. Wren,et al.  Real-Time 3-D Tracking of the Human Body , 1996 .

[23]  Colwyn Trevarthen,et al.  Sharing makes sense: intersubjectivity and the making of an infant’s meaning , 1987 .

[24]  D. McNeill,et al.  Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information , 1998 .

[25]  Leslie Pack Kaelbling,et al.  The Synthesis of Digital Machines With Provable Epistemic Properties , 1986, TARK.

[26]  Michael Grüninger,et al.  Introduction , 2002, CACM.

[27]  R. Krauss,et al.  Do conversational hand gestures communicate? , 1991, Journal of personality and social psychology.

[28]  Akikazu Takeuchi,et al.  Communicative facial displays as a new conversational modality , 1993, INTERCHI.

[29]  Kristinn R. Thórisson,et al.  The Power of a Nod and a Glance: Envelope Vs. Emotional Feedback in Animated Conversational Agents , 1999, Appl. Artif. Intell..

[30]  Justine Cassell,et al.  External manifestations of trustworthiness in the interface , 2000, CACM.

[31]  Clark Elliott,et al.  Autonomous Agents as Synthetic Characters , 1998, AI Mag..

[32]  W. Rogers,et al.  THE CONTRIBUTION OF KINESIC ILLUSTRATORS TOWARD THE COMPREHENSION OF VERBAL BEHAVIOR WITHIN UTTERANCES , 1978 .

[33]  Brian Scassellati,et al.  Alternative Essences of Intelligence , 1998, AAAI/IAAI.