A multimodal analysis of the sequential organization of verbal and nonverbal interaction

The paper addresses the study of the formal features of conversation from both a theoretical and an empirical perspective. In particular, following the framework and terminology of Sacks, Schlegoff and Jefferson (1974), turn-taking and sequencing are designated ‘formal’ features of interaction. Furthermore, the study adopts and fits into the framework of the theoretical modelling of human–computer interaction (HCI), as outlined in Hunyadi (2011). After reviewing the theoretical background of the multimodal analysis of conversation, a communicative act-based pragmatic annotation system will be outlined which is to contribute to a better understanding of the sequential organization of interaction. Finally, a case study on the sequential organization of nonverbal cues for turn regulation as well as their alignment with verbal content will be described.

[1]  John R. Searle,et al.  Speech Acts: An Essay in the Philosophy of Language , 1970 .

[2]  V. Yngve.  On getting a word in edgewise , 1970 .

[3]  S. Duncan,et al.  Some Signals and Rules for Taking Speaking Turns in Conversations , 1972 .

[4]  A. Kendon.  Some Relationships Between Body Motion and Speech , 1972 .

[5]  E. Schegloff,et al.  A simplest systematics for the organization of turn-taking for conversation , 1974 .

[6]  M. Knapp,et al.  Turn-Taking in Conversations. , 1975 .

[7]  C. Goodwin.  Conversational Organization: Interaction Between Speakers and Hearers , 1981 .

[8]  C. Creider.  Hand and Mind: What Gestures Reveal about Thought , 1994 .

[9]  Harvey Sacks,et al.  Lectures on Conversation , 1995 .

[10]  David J. Silverman,et al.  Harvey Sacks: Social Science and Conversation Analysis , 1998 .

[11]  J. Cassell,et al.  Turn taking vs. Discourse Structure: How Best to Model Multimodal Conversation , 1998 .

[12]  Paul ten Have,et al.  Doing Conversation Analysis: A Practical Guide , 1999 .

[13]  Jens Allwood,et al.  An activity-based approach to pragmatics , 2000, Abduction, Belief and Context in Dialogue.

[14]  Verschueren,et al.  语用学新解 : [英文版] = Understanding Pragmatics , 2000 .

[15]  Harry Bunt,et al.  The ABC of Computational Pragmatics , 2000, Abduction, Belief and Context in Dialogue.

[16]  Jacob L. Mey,et al.  Pragmatics: An Introduction , 2001 .

[17]  R. Gibson,et al.  What the Face Reveals , 2002 .

[18]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[19]  Mark Okrent.  Equipment, World, and Language , 2002 .

[20]  Paul Boersma,et al.  Praat: doing phonetics by computer , 2003 .

[21]  A. Kendon.  Gesture: Visible Action as Utterance , 2004 .

[22]  Hermann Kaindl,et al.  Modeling of interaction design by end users through discourse modeling , 2008, IUI '08.

[23]  Jack Sidnell,et al.  Conversation Analysis: List of tables , 2009 .

[24]  Csr Young,et al.  How to Do Things With Words , 2009 .

[25]  N. J. Enfield,et al.  The Anatomy of Meaning: Speech, Gesture, and Composite Utterances , 2009 .

[26]  Kôiti Hasida,et al.  Towards an ISO Standard for Dialogue Act Annotation , 2010, LREC.