On Factoring Out a Gesture Typology from the Bielefeld Speech-and-Gesture-Alignment Corpus (SAGA)

People communicate multimodally. Most prominently, they co-produce speech and gesture. How do they do that? Studying the interplay of both modalities has to be informed by empirically observed communication behavior. We present a corpus built of speech and gesture data gained in a controlled study. We describe 1) the setting underlying the data; 2) annotation of the data; 3) reliability evalution methods and results; and 4) applications of the corpus in the research domain of speech and gesture alignment.

[1]  M. Denis.  The description of routes : A cognitive approach to the production of spatial discourse , 1997 .

[2]  Raymond Reiter,et al.  A Logic for Default Reasoning , 1987, Artif. Intell..

[3]  Stefan Kopp,et al.  Regulating Dialogue with Gestures - Towards an Empirically Grounded Simulation with Conversational Agents , 2011, SIGDIAL Conference.

[4]  Martha W. Alibali,et al.  Gesture in Spatial Cognition: Expressing, Communicating, and Thinking About Spatial Information , 2005, Spatial Cogn. Comput..

[5]  Alexander Mehler,et al.  Representing and Maintaining Large Corpora , 2007 .

[6]  E. Schegloff,et al.  A simplest systematics for the organization of turn-taking for conversation , 1974 .

[7]  Ludwig Nagl,et al.  Charles Sanders Peirce , 1992 .

[8]  S. Goldin-Meadow,et al.  When Gestures and Words Speak Differently , 1997 .

[9]  D. McNeill.  Hand and Mind , 1995 .

[10]  Alexander Mehler,et al.  Taking Fingerprints of Speech-And-Gesture Ensembles. Approaching Empirical Evidence of Intrapersonal Alignment in Multimodal Communication , 2008 .

[11]  Stefan Kopp,et al.  Increasing the expressiveness of virtual agents: autonomous generation of speech and gesture for spatial description tasks , 2009, AAMAS.

[12]  W. Wundt.  Völkerpsychologie : eine Untersuchung der Entwicklungsgesetze von Sprache, Mythus und Sitte , 1909 .

[13]  M. Halliday.  NOTES ON TRANSITIVITY AND THEME IN ENGLISH. PART 2 , 1967 .

[14]  Stefan Kopp,et al.  MURML: A Multimodal Utterance Representation Markup Language for Conversational Agents , 2002 .

[15]  Stefan Kopp,et al.  Systematicity and Idiosyncrasy in Iconic Gesture Use: Empirical Analysis and Computational Modeling , 2009, Gesture Workshop.

[16]  Jürgen Streeck,et al.  Depicting by gesture , 2008 .

[17]  Aravind K. Joshi,et al.  Tree-Adjoining Grammars , 1997, Handbook of Formal Languages.

[18]  Isabella Poggi,et al.  From a Typology of Gestures to a Procedure for Gesture Production , 2001, Gesture Workshop.

[19]  P. Ekman,et al.  The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding , 1969 .

[20]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[21]  A. Kendon.  Gesticulation and Speech: Two Aspects of the Process of Utterance , 1981 .

[22]  Stefan Kopp,et al.  The Behavior Markup Language: Recent Developments and Challenges , 2007, IVA.

[23]  Stefan Kopp,et al.  Speech-Gesture Alignment , 2007 .

[24]  Charles Sanders Peirce,et al.  On a New List of Categories , 2006 .

[25]  Owen Rambow,et al.  Tree adjoining grammars : formalisms, linguistic analysis, and processing , 2000 .

[26]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[27]  Umberto Eco,et al.  A theory of semiotics , 1976, Advances in semiotics.

[28]  Annette Herskovits.  Language and Spatial Cognition: An Interdisciplinary Study of the Prepositions in English , 2009 .

[29]  Hannes Rieser.  Gestures Indicating Dialogue Structure , 2011 .

[30]  Stefan Kopp,et al.  MODELING THE PRODUCTION OF COVERBAL ICONIC GESTURES BY LEARNING BAYESIAN DECISION NETWORKS , 2010, Appl. Artif. Intell..

[31]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[32]  Stefan Kopp,et al.  Multimodal Communication from Multimodal Thinking - towards an Integrated Model of Speech and Gesture Production , 2008, Int. J. Semantic Comput..

[33]  Gabriele Paul,et al.  Approaches to abductive reasoning: an overview , 1993, Artificial Intelligence Review.

[34]  Bob Carpenter,et al.  The logic of typed feature structures , 1992 .

[35]  Stefan Kopp,et al.  GNetIc - Using Bayesian Decision Networks for Iconic Gesture Generation , 2009, IVA.

[36]  S. Kita.  Pointing: Where language, culture, and cognition meet , 2003 .

[37]  Matthew Stone,et al.  Microplanning with Communicative Intentions: The SPUD System , 2001, Comput. Intell..

[38]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .