Gesture, Prosody and Lexicon in Task-Oriented Dialogues: Multimedia Corpus Recording and Labelling

The aim of the DiaGest Project is to study interdependencies between gesture, lexicon, and prosody in Polish dialogues. The material under study comprises three tasks realised by twenty pairs of subjects. Two tasks involve instructional, task-oriented dialogues, while the third is based on a question answering procedure. A system for corpus labelling is currently being designed on the basis of current standards. The corpus will be annotated for gestures, lexical content of utterances, intonation and rhythm. In order to relate various phenomena to the contextualized meaning of dialogue utterances, the material will also be tagged in terms of dialogue acts. Synchronised tags will be placed in respective annotation tiers in ELAN. A number of detailed studies related to the problems of gesture-prosody, gesture-lexicon and prosody-lexicon interactions will be carried out on the basis of the tagged material.

[1]  Mark G. Core,et al.  Coding Dialogs with the DAMSL Annotation Scheme , 1997 .

[2]  Florian Schiel,et al.  Gestures During Overlapping Speech in multimodal HumanMachine Dialogues , 2001 .

[3]  E. Maier,et al.  Dialogue Acts in VERBMOBIL , 1995 .

[4]  Piet Mertens,et al.  The Prosogram: Semi-Automatic Transcription of Prosody Based on a Tonal Perception Model , 2004 .

[5]  Harry Bunt,et al.  Designing an Open, Multidimensional Dialogue Act Taxonomy , 2006 .

[6]  Norma C Mendoza-Denton,et al.  Structuring Information through Gesture and Intonation , 2005 .

[7]  Anita Wagner,et al.  Phonetics and its applications. Festschrift for Jens-Peter Koster , 2002 .

[8]  D. Bolinger.  INTONATION AND GESTURE , 1983 .

[9]  Costanza Navarretta,et al.  The MUMIN multimodal coding scheme , 2005 .

[10]  Ulrike Gut,et al.  Vocale - A Semi-Automatic Annotation Tool for Prosodic Research , 2002 .

[11]  Sotaro Kita,et al.  Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders , 1997, Gesture Workshop.

[12]  E. Barnard,et al.  Automatic intonation modeling with INTSINT , 2004 .

[13]  Roger K. Moore,et al.  Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation , 2000 .

[14]  Marion Klein,et al.  Standardisation Efforts on the Level of Dialogue Act in the MATE Project , 1998 .

[15]  Ipke Wachsmuth,et al.  Gesture and Sign Language in Human-Computer Interaction , 1998, Lecture Notes in Computer Science.

[16]  Marie-Francine Moens,et al.  Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation , 2000, Computational Linguistics.

[17]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[18]  Edward Gibson,et al.  A comparison of inter-transcriber reliability for two systems of prosodic annotation: rap (rhythm and pitch) and toBI (tones and break indices) , 2006, INTERSPEECH.

[19]  Daniel Hirst,et al.  Automatic modelling of fundamental frequency using a quadratic sline function , 1993 .

[20]  Adam Przepiórkowski,et al.  A Flexemic Tagset for Polish , 2003 .

[21]  Great Britain. Hm Factory Inspectorate.  An introductory guide , 1987 .

[22]  Emiel Krahmer,et al.  The Effects of Visual Beats on Prosodic Prominence: Acoustic Analyses, Auditory Perception and Visual Perception. , 2007 .

[23]  Craig Martell.  FORM: An Extensible, Kinematically-based Gesture Annotation Scheme , 2002, LREC.

[24]  Jean Carletta,et al.  HCRC dialogue structure coding manual , 1995 .

[25]  Rashid Ansari,et al.  Multimodal signal analysis of prosody and hand motion: Temporal correlation of speech and gestures , 2002, 2002 11th European Signal Processing Conference.

[26]  Daniel Hirst,et al.  Levels of Representation and Levels of Analysis for the Description of Intonation Systems , 2000 .

[27]  Alex Waibel,et al.  Intelligent animated agents for interactive language training , 1998, SIGC.

[28]  Michael Neff,et al.  An annotation scheme for conversational gestures: how to economically capture timing and form , 2007, Lang. Resour. Evaluation.

[29]  J. Burgoon,et al.  Nonverbal Communication , 2018, Encyclopedia of Evolutionary Psychological Science.

[30]  Michał Łesiów,et al.  Studia z językoznawstwa słowiańskiego , 1995 .

[31]  Ronald A. Cole,et al.  Perceptive animated interfaces: first steps toward a new paradigm for human-computer interaction , 2003, Proc. IEEE.

[32]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[33]  I. Marlien,et al.  Proceedings of Speech Prosody 2004, Nara, Japan , 2004 .

[34]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[35]  Merle Horne,et al.  Prosody: Theory and Experiment , 2000 .

[36]  Harry Bunt,et al.  A Framework for Dialogue Act Specication , 2005 .

[37]  Dafydd Gibbon,et al.  Spoken Language System Assessment , 1997 .