A Multi-dimensional Annotation Scheme for Behaviour in Dialogues

In this paper we introduce a multi-dimensional annotation scheme for emotional and behavioural assessment in dialogue summaries. To test the soundness both of the annotation scheme and corresponding guidelines, reliability studies with nine independent annotators were carried out. As an illustration of the utility of our scheme, we have applied it to an already published study and verified whether the same conclusions hold. We hope that, in using our scheme, researchers will be able to save a lot of time and effort that, otherwise, would be spent in planning, developing and testing a scheme of their own.

[1]  Renata Vieira,et al.  An Empirically-based System for Processing Definite Descriptions , 2000, CL.

[2]  J. Kooistra,et al.  An introduction to English syntax , 1933 .

[3]  Mirella Lapata,et al.  Constructing Corpora for the Development and Evaluation of Paraphrase Systems , 2008, CL.

[4]  Ron Artstein,et al.  Survey Article: Inter-Coder Agreement for Computational Linguistics , 2008, CL.

[5]  Brigitte Krenn,et al.  Fully generated scripted dialogue for embodied agents , 2008, Artif. Intell..

[6]  R. Craggs,et al.  A two dimensional annotation scheme for emotion in dialogue , 2004 .

[7]  Shogo Nishida,et al.  Social summarization of text feedback for online auctions and interactive presentation of the summary , 2006, IUI '06.

[8]  Paul Piwek,et al.  Politeness and Bias in Dialogue Summarization: Two Exploratory Studies , 2006, Computing Attitude and Affect in Text.

[9]  Sara Mills,et al.  Gender and Politeness , 2003 .

[10]  K. McKeown,et al.  Discourse Strategies for Generating Natural-Language Text , 1985, Artif. Intell..

[11]  S. Mills.  Gender and Politeness , 2003 .

[12]  Jeannett Martin.  Beyond Exchange : Appraisal Systems in English , 2000 .

[13]  Janyce Wiebe,et al.  Computing Attitude and Affect in Text: Theory and Applications , 2005, The Information Retrieval Series.

[14]  Elmar Nöth,et al.  The Recognition of Emotion , 2000 .

[15]  Bing Liu,et al.  Mining Opinion Features in Customer Reviews , 2004, AAAI.

[16]  Andrew Ortony,et al.  The Cognitive Structure of Emotions , 1988 .

[17]  Cecilia Ovesdotter Alm,et al.  Emotions from Text: Machine Learning for Text-based Emotion Prediction , 2005, HLT.

[18]  S. Hunston,et al.  Evaluation in Text: Authorial Stance and the Construction of Discourse , 2001 .

[19]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[20]  Douglas Walton,et al.  One-sided arguments : a dialectical analysis of bias , 1999 .

[21]  Michael L. Littman,et al.  Measuring praise and criticism: Inference of semantic orientation from association , 2003, TOIS.

[22]  Janyce Wiebe,et al.  Articles: Recognizing Contextual Polarity: An Exploration of Features for Phrase-Level Sentiment Analysis , 2009, CL.

[23]  Ellen Spertus,et al.  Smokey: Automatic Recognition of Hostile Messages , 1997, AAAI/IAAI.

[24]  Elena Lloret,et al.  Summarizing Threads in Blogs Using Opinion Polarity , 2009 .

[25]  Brian MacWhinney,et al.  Pragmatics in memory: A study of natural conversation , 1977 .