I Couldn't Agree More: The Role of Conversational Structure in Agreement and Disagreement Detection in Online Discussions

Determining when conversational participants agree or disagree is instrumental for broader conversational analysis; it is necessary, for example, in deciding when a group has reached consensus. In this paper, we describe three main contributions. We show how different aspects of conversational structure can be used to detect agreement and disagreement in discussion forums. In particular, we exploit information about meta-thread structure and accommodation between participants. Second, we demonstrate the impact of the features using 3-way classification, including sentences expressing disagreement, agreement or neither. Finally, we show how to use a naturally occurring data set with labels derived from the sides that participants choose in debates on createdebate.com. The resulting new agreement corpus, Agreement by Create Debaters (ABCD) is 25 times larger than any prior corpus. We demonstrate that using this data enables us to outperform the same system trained on prior existing in-domain smaller annotated datasets.

[1]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[2]  Mari Ostendorf,et al.  Detection Of Agreement vs. Disagreement In Meetings: Training With Unlabeled Data , 2003, NAACL.

[3]  Claire Cardie,et al.  Improving Agreement and Disagreement Identification in Online Discussions with A Socially-Tuned Sentiment Lexicon , 2014, WASSA@ACL.

[4]  Dan Klein,et al.  Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network , 2003, NAACL.

[5]  Amita Misra,et al.  Topic Independent Identification of Agreement and Disagreement in Social Media Dialogue , 2013, SIGDIAL Conference.

[6]  Alexander S. Yeh,et al.  More accurate tests for the statistical significance of result differences , 2000, COLING.

[7]  Arjun Mukherjee,et al.  Analysis of Linguistic Style Accommodation in Online Debates , 2012, COLING.

[8]  Kristin Precoda,et al.  Detection of Agreement and Disagreement in Broadcast Conversations , 2011, ACL.

[9]  Russell V. Lenth,et al.  Computer Intensive Methods for Testing Hypotheses: An Introduction , 1990 .

[10]  Kristin Precoda,et al.  Identifying Agreement/Disagreement in Conversational Speech: A Cross-Lingual Study , 2011, INTERSPEECH.

[11]  Jacob Andreas,et al.  Annotating Agreement and Disagreement in Threaded Discussion , 2012, LREC.

[12]  Jacob Andreas,et al.  Detecting Influencers in Written Online Conversations , 2012 .

[13]  Julia Hirschberg,et al.  Identifying Agreement and Disagreement in Conversational Speech: Use of Bayesian Networks to Model Pragmatic Dependencies , 2004, ACL.

[14]  Kathy McKeown,et al.  Columbia NLP: Sentiment Detection of Sentences and Subjective Phrases in Social Media , 2014, *SEMEVAL.

[15]  Richard E. Ladner,et al.  Agreement/Disagreement Classification: Exploiting Unlabeled Data using Contrast Classifiers , 2006, HLT-NAACL.

[16]  H. Giles,et al.  Accommodation theory: Communication, context, and consequence. , 1991 .

[17]  Andrew Y. Ng,et al.  Parsing with Compositional Vector Grammars , 2013, ACL.

[18]  Wei-Hao Lin,et al.  Which Side are You on? Identifying Perspectives at the Document and Sentence Levels , 2006, CoNLL.

[19]  Jon M. Kleinberg,et al.  Echoes of power: language effects and power differences in social interaction , 2011, WWW.

[20]  Fadi Biadsy,et al.  Contextual Phrase-Level Polarity Analysis Using Lexical Affect Scoring and Syntactic N-Grams , 2009, EACL.

[21]  Arjun Mukherjee,et al.  Discovering User Interactions in Ideological Discussions , 2013, ACL.

[22]  Paul Thomas,et al.  Unifying Local and Global Agreement and Disagreement Classification in Online Debates , 2012, WASSA@ACL.

[23]  Bernd Opitz,et al.  Bootstrapping an Unsupervised Approach for Classifying Agreement and Disagreement , 2013, NODALIDA.

[24]  Dragomir R. Radev,et al.  Detecting Subgroups in Online Discussions by Modeling Positive and Negative Relations among Participants , 2012, EMNLP.

[25]  Jean Carletta,et al.  The AMI meeting corpus , 2005 .

[26]  Dragomir R. Radev,et al.  Subgroup Detection in Ideological Discussions , 2012, ACL.

[27]  Weiwei Guo,et al.  Modeling Sentences in the Latent Space , 2012, ACL.

[28]  Theresa Wilson,et al.  Agreement detection in multiparty conversation , 2009, ICMI-MLMI '09.

[29]  Jonathan T. Morgan,et al.  Annotating Social Acts: Authority Claims and Alignment Moves in Wikipedia Talk Pages , 2011 .

[30]  M. Walker,et al.  How can you say such things?!?: Recognizing Disagreement in Informal Political Argument , 2011 .

[31]  Marilyn A. Walker,et al.  A Corpus for Research on Deliberation and Debate , 2012, LREC.

[32]  Matt Thomas,et al.  Get out the vote: Determining support or opposition from Congressional floor-debate transcripts , 2006, EMNLP.

[33]  Andreas Stolcke,et al.  The ICSI Meeting Corpus , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..