A Large-scale Dataset for Argument Quality Ranking: Construction and Analysis

Identifying the quality of free-text arguments has become an important task in the rapidly expanding field of computational argumentation. In this work, we explore the challenging task of argument quality ranking. To this end, we created a corpus of 30,497 arguments carefully annotated for point-wise quality, released as part of this work. To the best of our knowledge, this is the largest dataset annotated for point-wise argument quality, larger by a factor of five than previously released datasets. Moreover, we address the core issue of inducing a labeled score from crowd annotations by performing a comprehensive evaluation of different approaches to this problem. In addition, we analyze the quality dimensions that characterize this dataset. Finally, we present a neural method for argument quality ranking, which outperforms several baselines on our own dataset, as well as previous methods published for another dataset.

[1]  Michal Jacovi,et al.  Automatic Argument Quality Assessment - New Datasets and Methods , 2019, EMNLP.

[2]  Paolo Torroni,et al.  Argumentation Mining , 2016, ACM Trans. Internet Techn..

[3]  Aristotle,et al.  On rhetoric : a theory of civic discourse , 1993 .

[4]  Noam Slonim,et al.  Towards an argumentative content search engine using weak supervision , 2018, COLING.

[5]  Iryna Gurevych,et al.  Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM , 2016, ACL.

[6]  Yvette Graham,et al.  Improving Evaluation of Machine Translation Quality Estimation , 2015, ACL.

[7]  Trevor J. M. Bench-Capon,et al.  Altruism and agents: an argumentation based approach to designing agent decision mechanisms , 2009, AAMAS.

[8]  Benno Stein,et al.  Building an Argument Search Engine for the Web , 2017, ArgMining@EMNLP.

[9]  Benno Stein,et al.  Computational Argumentation Quality Assessment in Natural Language , 2017, EACL.

[10]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[11]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[12]  Eyal Shnarch,et al.  Are You Convinced? Choosing the More Convincing Evidence with a Siamese Network , 2019, ACL.

[13]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[14]  Eyal Shnarch,et al.  Corpus Wide Argument Mining - a Working Solution , 2019, AAAI.

[15]  Iryna Gurevych,et al.  Finding Convincing Arguments Using Scalable Bayesian Preference Learning , 2018, TACL.

[16]  Chris Reed.  Proceedings of the Third Workshop on Argument Mining (ArgMining2016) , 2016 .

[17]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[18]  Benno Stein,et al.  The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants , 2017, NAACL.

[19]  Anna Rumshisky,et al.  Length, Interchangeability, and External Knowledge: Observations from Predicting Argument Convincingness , 2017, IJCNLP.

[20]  Iryna Gurevych,et al.  Annotating Argument Components and Relations in Persuasive Essays , 2014, COLING.

[21]  J. H. Steiger.  Tests for comparing elements of a correlation matrix. , 1980 .

[22]  W. Bialek,et al.  Information-based clustering. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[23]  Iryna Gurevych,et al.  Classification and Clustering of Arguments with Contextualized Word Embeddings , 2019, ACL.

[24]  Brian Ecker,et al.  Argument Mining: Extracting Arguments from Online Dialogue , 2015, SIGDIAL Conference.

[25]  Claire Cardie,et al.  The Role of Pragmatic and Discourse Context in Determining Argument Impact , 2019, EMNLP.

[26]  Timothy Baldwin,et al.  Testing for Significance of Increased Correlation with Human Judgment , 2014, EMNLP.

[27]  Timothy J. Hazen,et al.  Ranking Passages for Argument Convincingness , 2019, ArgMining@ACL.

[28]  Indrajit Bhattacharya,et al.  Stance Classification of Context-Dependent Claims , 2017, EACL.

[29]  Chris Reed,et al.  Argumentation Schemes , 2008 .

[30]  Ch. Perelman,et al.  The New Rhetoric: A Treatise on Argumentation , 1971 .

[31]  John A. Hartigan,et al.  Clustering Algorithms , 1975 .

[32]  Dirk Hovy,et al.  Learning Whom to Trust with MACE , 2013, NAACL.