The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena

This paper deals with a multimodal annotation scheme dedicated to the study of gestures in interpersonal communication, with particular regard to the role played by multimodal expressions for feedback, turn management and sequencing. The scheme has been developed under the framework of the MUMIN network and tested on the analysis of multimodal behaviour in short video clips in Swedish, Finnish and Danish. The preliminary results obtained in these studies show that the reliability of the categories defined in the scheme is acceptable, and that the scheme as a whole constitutes a versatile analysis tool for the study of multimodal communication behaviour.

[2]  Roddy Cowie,et al.  Describing the emotional states that are expressed in speech , 2003, Speech Commun..

[3]  Mary McGee Wood,et al.  A Categorical Annotation Scheme for Emotion in the Linguistic Content of Dialogue , 2004, ADS.

[4]  J. Allwood,et al.  A study of gestural feedback expressions , 2006 .

[5]  J. Harrigan,et al.  Proxemics, kinesics, and gaze. , 2005 .

[6]  Jens Allwood.  Dialog Coding - Function and Grammar: Göteborg Coding Schemsas , 2001 .

[7]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[8]  Daniel Thalmann,et al.  Multimodal Authoring Tool for Populating a Database of Emotional Reactive Animations , 2005, MLMI.

[9]  Jens Allwood,et al.  The structure of dialog , 1999 .

[10]  Loredana Cerrato.  Some characteristics of feedback expressions in Swedish , 2007 .

[11]  Gérard Bailly,et al.  Degrees of freedom of facial movements in face-to-face conversational speech , 2006 .

[12]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[13]  Loredana Cerrato,et al.  Investigating Communicative Feedback Phenomena across Languages and Modalities , 2007 .

[14]  Donald W. Fiske,et al.  Face-to-face interaction: Research, methods, and theory , 1977 .

[15]  Jean-Claude Martin,et al.  Annotation of Emotions in Real-Life Video Interviews: Variability between Coders , 2006, LREC.

[16]  D. Bouwhuis,et al.  The Structure of Multimodal Dialogue , 1989 .

[17]  P. Ekman,et al.  Unmasking the face : a guide to recognizing emotions from facial clues , 1975 .

[18]  Herbert H. Clark,et al.  Contributing to Discourse , 1989, Cogn. Sci..

[19]  Niels Ole Bernsen,et al.  THE NITE WORKBENCH. A Tool for Annotation of Natural Interactivity and Multimodal Data , 2002, LREC.

[20]  K. Scherer,et al.  The New Handbook of Methods in Nonverbal Behavior Research , 2008 .

[21]  Joakim Nivre,et al.  On the Semantics and Pragmatics of Linguistic Feedback , 1992, J. Semant..

[22]  Costanza Navarretta,et al.  The MUMIN multimodal coding scheme , 2005 .

[23]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[24]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[25]  Charles S. Peirce,et al.  Elements of logic , 1960 .

[26]  K. Scherer.  What are emotions? And how can they be measured? , 2005 .

[27]  J. Cassell,et al.  Nudge nudge wink wink: elements of face-to-face conversation for embodied conversational agents , 2001 .

[28]  T. Dalgleish.  Basic Emotions , 2004 .

[29]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[30]  W.P.M.S. Spooren,et al.  The use of discourse data in language use research , 2004 .

[31]  Toni Rietveld,et al.  Statistical Techniques for the Study of Language and Language Behaviour , 1993 .

[32]  Gwyneth Doherty-Sneddon,et al.  The Reliability of a Dialogue Structure Coding Scheme , 1997, CL.

[33]  T. Dalgleish,et al.  Handbook of cognition and emotion , 1999 .

[34]  Teresa Sikorski.  Improving Dialogue Annotation Reliability , 1998 .

[35]  Kristinn R. Thórisson,et al.  Natural Turn-Taking Needs No Manual: Computational Theory and Model, from Perception to Action , 2002 .

[36]  Florian Schiel,et al.  Development of the UserState Conventions for the Multimodal Corpus in SmartKom , 2002 .