Towards Experimental Specification and Evaluation of Lifelike Multimodal Behavior

ABSTRACT In this paper we introduce theLimsi Embodied Agent project which tackles the following issues of Embodied ConversationalAgent (ECA) specification and evaluation: the need to groundECA’s behavior on video-taped annotations of applicationdependent human behavior, the granularity of the language forspecifying the ECAmultimodal behavior, and the evaluation of the use of ECA in Human-Computer Interaction. In this paper,we describe preliminary work and future directions in each ofthese issues. Categories and Subject Descriptors H.5.2-H.5.1 Information Interfaces and Presentation[ ]: UserInterface – interaction styles, standardization, ergonomics, userinterface management systems. Multimedia Information Systems– evaluation/methodology. General Terms Design, Experimentation, Human Factors, Standardization. Keywords Multimodal interaction and integration,multimodal coding scheme. 1. INTRODUCTION There is still a lack of appropriate and global answers to thequestion of the “natural” behavior of Embodied ConversationalAgent (ECA). The specification of multimodal behavior of ECAis often based on knowledge extracted from the literature inseveral domains such as Psychology, Sociology and Linguistics.As partly suggested by [14] [6], we believe that in order to belifelike, multimodal behavior of agents needs to be grounded onexperimental studies in the same application context (i.e. themultimodal behavior of pedagogical ECA should be based onvideo recording and annotation of teacher’s behavior in“similar” settings). In this paper, we describe how we intend touse such an experimental approach with theLimsi Embodied Agent (LEA). But how do we go from annotating humanmultimodal behavior to specifying the behavior of an ECA?Existing specification languages are mostly dedicated either tolow-level monomodal specification (i.e. angry facial expression)or toamodal “higher” level specifications which are translatedinto monomodal features (i.e. angry behavior generating facialexpression, intonation, gaze…). In the LEA project, we definean intermediate level of specification based on types ofcooperation between communicative modalities which can beuseful for fine-grain specification and evaluation ofmultimodal communicative behavior based on video corpus annotation [20].Finally, we describe our global methodological frameworkwhich can be considered as a checklist for defining theevaluation process of ECAs.