SemEval-2012 Task 7: Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

SemEval-2012 Task 7 presented a deceptively simple challenge: given an English sentence as a premise, select the sentence amongst two alternatives that more plausibly has a causal relation to the premise. In this paper, we describe the development of this task and its motivation. We describe the two systems that competed in this task as part of SemEval-2012, and compare their results to those achieved in previously published research. We discuss the characteristics that make this task so difficult, and offer our thoughts on how progress can be made in the future.

[1]  L. R. Dice.  Measures of the Amount of Ecologic Association Between Species , 1945 .

[2]  H. Hart,et al.  Causation in the Law , 1961 .

[3]  G. Williams.  Causation in the Law , 1961, The Cambridge Law Journal.

[4]  J. Jenkins,et al.  Word association norms , 1964 .

[5]  E. Pols.  Causal Powers: A Theory of Natural Necessity , 1976 .

[6]  A. Müller.  The Cement of the Universe: A Study of Causation , 1977 .

[7]  T. Shultz.  Rules of Causal Attribution. , 1982 .

[8]  T. Trabasso.  Causal Cohesion and Story Coherence. , 1982 .

[9]  J. Woodward,et al.  Scientific Explanation and the Causal Structure of the World , 1988 .

[10]  Elizabeth B. Bernhardt,et al.  Learning and comprehension of text , 1988 .

[11]  Scott M. Smith,et al.  Computer Intensive Methods for Testing Hypotheses: An Introduction , 1989 .

[12]  Kenneth Ward Church,et al.  Word Association Norms, Mutual Information, and Lexicography , 1989, ACL.

[13]  T. Trabasso,et al.  Logical necessity and transitivity of causal relations in stories , 1989 .

[14]  Yoav Shoham,et al.  Nonmonotonic Reasoning and Causation , 1990, Cogn. Sci..

[15]  L. Keita.  Scientific Explanation and the Causal Structure of the World , 1990 .

[16]  M. Singer,et al.  Validation of causal bridging inferences in discourse understanding. , 1992 .

[17]  D. Medin,et al.  The role of covariation versus mechanism information in causal attribution , 1995, Cognition.

[18]  Douglas B. Lenat,et al.  CYC: a large-scale investment in knowledge infrastructure , 1995, CACM.

[19]  V. Thompson.  CONDITIONAL REASONING: THE NECESSARY AND SUFFICIENT CONDITIONS , 1995 .

[20]  J. McCarthy.  ELABORATION TOLERANCE , 1997 .

[21]  Grigoris Antoniou,et al.  Nonmonotonic reasoning , 1997 .

[22]  V. Lifschitz.  Cracking an Egg: An Exercise in Commonsense Reasoning , 1998 .

[23]  Daniel Marcu,et al.  A Decision-Based Approach to Rhetorical Parsing , 1999, ACL.

[24]  Leora Morgenstern,et al.  Mid-Sized Axiomatizations of Commonsense Problems: A Case Study in Egg Cracking , 2001, Stud Logica.

[25]  Peter D. Turney.  Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL , 2001, ECML.

[26]  Murray Shanahan,et al.  An attempt to formalise a non-trivial benchmark problem in common sense reasoning , 2004, Artif. Intell..

[27]  Leora Morgenstern,et al.  Introduction: Progress in formal commonsense reasoning , 2004, Artif. Intell..

[28]  Hugo Liu,et al.  ConceptNet — A Practical Commonsense Reasoning Tool-Kit , 2004 .

[29]  L. Morgenstern.  A First-Order Axiomatization of the Surprise Birthday Present Problem ( Preliminary Report ) : Abbreviated Version , 2005 .

[30]  Jerry R. Hobbs.  Toward a Useful Concept of Causality for Lexical Semantics , 2005, J. Semant..

[31]  Ido Dagan,et al.  The Third PASCAL Recognizing Textual Entailment Challenge , 2007, ACL-PASCAL@ACL.

[32]  Henry Lieberman,et al.  AnalogySpace: Reducing the Dimensionality of Common Sense Knowledge , 2008, AAAI.

[33]  Kenji Sagae,et al.  Analysis of Discourse Structure with Syntactic Dependencies and Data-Driven Shift-Reduce Parsing , 2009, IWPT.

[34]  R. Swanson,et al.  Identifying Personal Stories in Millions of Weblog Entries , 2009, ICWSM 2009.

[35]  Lenhart K. Schubert,et al.  Learning from the Web: Extracting General World Knowledge from Noisy Text , 2010, Collaboratively-Built Knowledge Sources and AI.

[36]  Andrew S. Gordon,et al.  Open-domain Commonsense Reasoning Using Discourse Relations from a Corpus of Weblog Stories , 2010, HLT-NAACL 2010.

[37]  Sung-Hyon Myaeng,et al.  Automatic construction of a large-scale situation ontology by mining how-to instructions from the web , 2010, J. Web Semant..

[38]  Cosmin Adrian Bejan,et al.  Commonsense Causal Reasoning Using Millions of Personal Stories , 2011, AAAI.

[39]  Sanda M. Harabagiu,et al.  UTDHLT: COPACETIC System for Choosing Plausible Alternatives , 2012, *SEMEVAL.