The SAMMIE Corpus of Multimodal Dialogues with an MP3 Player

We describe a corpus of multimodal dialogues with an MP3player collected in Wizard-of-Oz experiments and annotated with a richfeature set at several layers. We are using the Nite XML Toolkit (NXT) to represent and further process the data. We designed an NXTdata model, converted experiment log file data and manualtranscriptions into NXT, and are building tools for additionalannotation using NXT libraries. The annotated corpus will be used to (i) investigate various aspects of multimodal presentation andinteraction strategies both within and across annotation layers; (ii) design an initial policy for reinforcement learning of multimodalclarification requests.

[1]  Massimo Poesio,et al.  Annotating a Corpus to Develop and Evaluate Discourse Entity Realization Algorithms: Issues and Preliminary Results , 2000, LREC.

[2]  Mark G. Core,et al.  Coding Dialogs with the DAMSL Annotation Scheme , 1997 .

[3]  Jan Alexandersson,et al.  Amigram-a general-purpose tool for multimodal corpus annotation , 2005 .

[4]  Massimo Poesio,et al.  The MATE/GNOME Proposals for Anaphoric Annotation, Revisited , 2004, SIGDIAL Workshop.

[5]  Douglas B. Moran,et al.  The Open Agent Architecture: A Framework for Building Distributed Software Systems , 1999, Appl. Artif. Intell..

[6]  Verena Rieser,et al.  An Experiment Setup for Collecting Data for Adaptive Output Planning in a Multimodal Dialogue System , 2005, ENLG.

[7]  Stefan Evert,et al.  The NITE XML Toolkit: Flexible annotation for multimodal language data , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[8]  Marilyn A. Walker,et al.  Learning Content Selection Rules for Generating Object Descriptions in Dialogue , 2005, J. Artif. Intell. Res..

[9]  Gwyneth Doherty-Sneddon,et al.  The Reliability of a Dialogue Structure Coding Scheme , 1997, CL.

[10]  Oliver Lemon,et al.  A Corpus Collection and Annotation Framework for Learning Multimodal Clarification Strategies , 2005, SIGDIAL.

[11]  David R. Traum,et al.  CONVERSATION ACTS IN TASK‐ORIENTED SPOKEN DIALOGUE , 1992, Comput. Intell..

[12]  Marilyn A. Walker,et al.  DATE: A Dialogue Act Tagging Scheme for Evaluation of Spoken Dialogue Systems , 2001, HLT.

[13]  David Traum,et al.  The Information State Approach to Dialogue Management , 2003 .

[14]  Harry Bunt,et al.  A Framework for Dialogue Act Specication , 2005 .