Argument Search: Assessing Argument Relevance

We report on the first user study on assessing argument relevance. Based on a search among more than 300,000 arguments, four standard retrieval models are compared on 40 topics for 20 controversial issues: every issue has one topic with a biased stance and another neutral one. Following TREC, the top results of the different models on a topic were pooled and relevance-judged by one assessor per topic. The assessors also judged the arguments' rhetorical, logical, and dialectical quality, the results of which were cross-referenced with the relevance judgments. Furthermore, the assessors were asked for their personal opinion, and whether it matched the predefined stance of a topic. Among other results, we find that Terrier's implementations of DirichletLM and DPH are on par, significantly outperforming TFIDF and BM25. The judgments of relevance and quality hardly correlate, giving rise to a more diverse set of ranking criteria than relevance alone. We did not measure a significant bias of assessors when their stance is at odds with a topic's stance.

[1]  Matthias Hagen,et al.  Webis at TREC 2018: Common Core Track , 2018, TREC.

[2]  E. Krabbe,et al.  Groundwork in the Theory of Argumentation , 2012 .

[3]  Robert Trapp,et al.  Perspectives on argumentation : essays in honor of Wayne Brockriede , 1990 .

[4]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[5]  Ralph H. Johnson.  Revisiting the Logical/Dialectical/Rhetorical Triumvirate , 2009 .

[6]  Sally Jackson,et al.  Design Thinking in Argumentation Theory and Practice , 2015 .

[7]  Mitesh M. Khapra,et al.  Show Me Your Evidence - an Automatic Method for Context Dependent Evidence Detection , 2015, EMNLP.

[8]  Gerald Albaum,et al.  The Likert Scale Revisited , 1997 .

[9]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[10]  Iryna Gurevych,et al.  Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM , 2016, ACL.

[11]  Douglas Walton,et al.  Fundamentals of critical argumentation , 2006, Critical reasoning and argumentation.

[12]  Vincent Ng,et al.  Modeling Argument Strength in Student Essays , 2015, ACL.

[13]  Benno Stein,et al.  Building an Argument Search Engine for the Web , 2017, ArgMining@EMNLP.

[14]  H. Rittel,et al.  Dilemmas in a general theory of planning , 1973 .

[15]  Iryna Gurevych,et al.  ArgumenText: Searching for Arguments in Heterogeneous Sources , 2018, NAACL.

[16]  Dima Mohammed,et al.  Goals in Argumentation: A Proposal for the Analysis and Evaluation of Public Political Arguments , 2016 .

[17]  Benno Stein,et al.  “PageRank” for Argument Relevance , 2017, EACL.

[18]  Benno Stein,et al.  Computational Argumentation Quality Assessment in Natural Language , 2017, EACL.

[19]  Gianni Amati,et al.  Frequentist and Bayesian Approach to Information Retrieval , 2006, ECIR.

[20]  Ralph H. Johnson.  Manifest Rationality: A Pragmatic Theory of Argument , 2000 .

[21]  Christopher W. Tindale,et al.  Groundwork in the Theory of Argumentation : Selected Papers of J . , 2014 .

[22]  John D. Lafferty,et al.  A study of smoothing methods for language models applied to Ad Hoc information retrieval , 2001, SIGIR '01.

[23]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[24]  F. H. Eemeren,et al.  Speech acts in argumentative discussions , 1984 .

[25]  Ben He,et al.  Terrier : A High Performance and Scalable Information Retrieval Platform , 2022 .

[26]  J. Anthony Blair,et al.  Rhetoric, Dialectic, and Logic as Related to Argument , 2012 .

[27]  F. C. P. Motta.  The theory of communicative action , 1991 .