Evaluating Users' Reactions to Human-Like Interfaces

An increasing number of dialogue systems are deployed to provide public services in our everyday lives. They are becoming more service-minded and several of them provide different channels for interaction. The rationale is to make automatic services available in new environments and more attractive to use. From a developer perspective, this affects the complexity of the requirements elicitation activity, as new combinations and variations in end-user interaction need to be considered. The aim of our investigation is to propose new parameters and metrics to evaluate multimodal dialogue systems endowed with embodied conversational agents (ECAs). These new matrics focus on the users, rather than on the system. Our assumption is that the intentional use of prosodic variation and the production of communicative nonverbal behaviour by users can give an indication of their attitude towards the system and might also help to evaluate the users' overall experience of the interaction. To test our hypothesis we carried out analyses on different Swedish corpora of interactions between users and multimodal dialogue systems. We analysed the prosodic variation in the way the users ended their interactions with the system and we observed the prodution of non-verbal communicative expressions by users. Our study supports the idea that the observation of users' prosodic variation and production of communicative non-verbal behaviour during the interaction with dialogue syslems could be used as an indication of whether or not the users are satisfied with the system performance.

[1]  Marilyn A. Walker,et al.  DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems , 2001, HLT.

[2]  E. Schegloff,et al.  Opening up Closings , 1973 .

[3]  Volker Strom,et al.  Visual prosody: facial movements accompanying speech , 2002, Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition.

[4]  Kristinn R. Thórisson.  Gandalf: an embodied humanoid capable of real-time multimodal dialogue with people , 1997, AGENTS '97.

[5]  Mark Steedman,et al.  Generating Facial Expressions for Speech , 1996, Cogn. Sci..

[6]  Joakim Gustafson,et al.  The august spoken dialogue system , 1999, EUROSPEECH.

[7]  Roxane Bertrand,et al.  About the relationship between eyebrow movements and Fo variations , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[8]  A. Damasio.  Descartes’ Error. Emotion, Reason and the Human Brain. New York (Grosset/Putnam) 1994. , 1994 .

[9]  Rosalind W. Picard.  Affective Computing , 1997 .

[10]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[11]  Dafydd Gibbon,et al.  Handbook of Multimodal and Spoken Dialogue Systems , 2000 .

[12]  Jens Edlund,et al.  Adapt - a multimodal conversational dialogue system in an apartment domain , 2000, INTERSPEECH.

[13]  Björn Granström,et al.  Measurements of articulatory variation and communicative signals in expressive speech , 2003, AVSP.

[14]  Michael F. McTear,et al.  Book Review: Spoken Dialogue Technology: Toward the Conversational User Interface, by Michael F. McTear , 2002, CL.

[15]  Jonas Beskow,et al.  Wavesurfer - an open source speech tool , 2000, INTERSPEECH.

[16]  Dominic W. Massaro,et al.  MULTIMODAL EMOTION PERCEPTION: ANALOGOUS TO SPEECH PROCESSES , 2000 .

[17]  P. Ekman.  Facial expression and emotion. , 1993, The American psychologist.

[18]  Anna Hjalmarsson.  Evaluating AdApt, a multi-modal conversational, dialogue system, using PARADISE , 2003 .

[19]  Jens Edlund,et al.  Turn-taking gestures and hour-glasses in a multi-modal dialogue system , 2002 .

[20]  Rudolf von Laban,et al.  The language of movement;: A guidebook to choreutics, , 1974 .

[21]  Marilyn A. Walker,et al.  Towards developing general models of usability with PARADISE , 2000, Natural Language Engineering.

[22]  J. Beskow.  Talking Heads - Models and Applications for Multimodal Speech Synthesis , 2003 .

[23]  Gregory A. Sanders,et al.  Measurement and evaluation of embodied conversational agents , 2001 .

[24]  Richard Lippmann,et al.  Speech recognition by machines and humans , 1997, Speech Commun..

[25]  Jonas Beskow,et al.  Developing and evaluating conversational agents , 2001 .

[26]  Catherine Pelachaud,et al.  Emotional Meaning and Expression in Animated Faces , 1999, IWAI.

[27]  Ricardo de Córdoba Herralde,et al.  The Role of Pitch and Tempo in Spanish Emotional Speech , 2002 .

[28]  J. Cassell,et al.  Embodied conversational agents , 2000 .