A Review of Winograd Schema Challenge Datasets and Approaches

The Winograd Schema Challenge is both a commonsense reasoning and natural language understanding challenge, introduced as an alternative to the Turing test. A Winograd schema is a pair of sentences differing in one or two words with a highly ambiguous pronoun, resolved differently in the two sentences, that appears to require commonsense knowledge to be resolved correctly. The examples were designed to be easily solvable by humans but difficult for machines, in principle requiring a deep understanding of the content of the text and the situation it describes. This paper reviews existing Winograd Schema Challenge benchmark datasets and approaches that have been published since its introduction.

[1]  Mitchell P. Marcus,et al.  OntoNotes: The 90% Solution , 2006, NAACL.

[2]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[3]  Arthur B. Markman,et al.  Knowledge Representation , 1998 .

[4]  Peter Szolovits,et al.  What Is a Knowledge Representation? , 1993, AI Mag..

[5]  Anette Frank,et al.  Addressing the Winograd Schema Challenge as a Sequence Ranking Task , 2018 .

[6]  Itamar Arel,et al.  Beyond the Turing Test , 2009, Computer.

[7]  Chitta Baral,et al.  Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge , 2019, ACL.

[8]  Tassilo Klein,et al.  Attention Is (not) All You Need for Commonsense Reasoning , 2019, ACL.

[9]  Quoc V. Le,et al.  A Simple Method for Commonsense Reasoning , 2018, ArXiv.

[10]  Olga Seminck,et al.  A Google-Proof Collection of French Winograd Schemas , 2017 .

[11]  Leora Morgenstern,et al.  Planning, Executing, and Evaluating the Winograd Schema Challenge , 2016, AI Mag..

[12]  Leora Morgenstern,et al.  The First Winograd Schema Challenge at IJCAI-16 , 2017, AI Mag..

[13]  Yejin Choi,et al.  WINOGRANDE: An Adversarial Winograd Schema Challenge at Scale , 2020, AAAI.

[14]  Chitta Baral,et al.  Towards Addressing the Winograd Schema Challenge - Building and Using a Semantic Parser and a Knowledge Hunting Module , 2015, IJCAI.

[15]  Hector J. Levesque,et al.  The Winograd Schema Challenge , 2011, AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.

[16]  Zhen-Hua Ling,et al.  Exploring Unsupervised Pretraining and Sentence Structure Modelling for Winograd Schema Challenge , 2019, ArXiv.

[17]  Chitta Baral,et al.  Knowledge Representation, Reasoning and Declarative Problem Solving , 2003 .

[18]  Gary Marcus,et al.  The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence , 2020, ArXiv.

[19]  V. Kaul,et al.  Planning , 2012 .

[20]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[21]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[22]  Arpit Sharma.  Using Answer Set Programming for Commonsense Reasoning in the Winograd Schema Challenge , 2019, Theory Pract. Log. Program..

[23]  Jackie Chi Kit Cheung,et al.  A Knowledge Hunting Framework for Common Sense Reasoning , 2018, EMNLP.

[24]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[25]  Catherine Havasi,et al.  Representing General Relational Knowledge in ConceptNet 5 , 2012, LREC.

[26]  Zhen-Hua Ling,et al.  Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models , 2019, ArXiv.

[27]  Johannes Fähndrich,et al.  A Marker Passing Approach to Winograd Schemas , 2018, JIST.

[28]  Yu Hu,et al.  Cause-Effect Knowledge Acquisition and Neural Association Model for Solving A Set of Winograd Schema Problems , 2017, IJCAI.

[29]  Yu Hu,et al.  Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems , 2017, AAAI Spring Symposia.

[30]  Yangqiu Song,et al.  A Distributed Solution for Winograd Schema Challenge , 2018, ICMLC.

[31]  Thomas Lukasiewicz,et al.  A Surprisingly Robust Trick for the Winograd Schema Challenge , 2019, ACL.

[32]  Thomas Lukasiewicz,et al.  WikiCREM: A Large Unsupervised Corpus for Coreference Resolution , 2019, EMNLP.

[33]  Ilya Sutskever,et al.  Language Models are Unsupervised Multitask Learners , 2019 .

[34]  Terry Winograd,et al.  Understanding natural language , 1974 .

[35]  Vincent Ng,et al.  Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge , 2012, EMNLP.

[36]  Jackie Chi Kit Cheung,et al.  On the Evaluation of Common-Sense Reasoning in Natural Language Understanding , 2018, ArXiv.

[37]  Lise Getoor,et al.  A short introduction to probabilistic soft logic , 2012, NIPS 2012.

[38]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[39]  Xiaodong Liu,et al.  A Hybrid Neural Network Model for Commonsense Reasoning , 2019, EMNLP.

[40]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[41]  Loizos Michael,et al.  Tackling the Winograd Schema Challenge Through Machine Logical Inferences , 2016, STAIRS.

[42]  Xiaodong Liu,et al.  Unsupervised Deep Structured Semantic Models for Commonsense Reasoning , 2019, NAACL.

[43]  Omer Levy,et al.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems , 2019, NeurIPS.

[44]  Rachel Rudinger,et al.  Gender Bias in Coreference Resolution , 2018, NAACL.

[45]  Jieyu Zhao,et al.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods , 2018, NAACL.

[46]  Loizos Michael,et al.  WinoFlexi: A Crowdsourcing Platform for the Development of Winograd Schemas , 2019, Australasian Conference on Artificial Intelligence.