Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge

Winograd Schema Challenge (WSC) is a pronoun resolution task which seems to require reasoning with commonsense knowledge. The needed knowledge is not present in the given text. Automatic extraction of the needed knowledge is a bottleneck in solving the challenge. The existing state-of-the-art approach uses the knowledge embedded in their pre-trained language model. However, the language models only embed part of the knowledge, the ones related to frequently co-existing concepts. This limits the performance of such models on the WSC problems. In this work, we build-up on the language model based methods and augment them with a commonsense knowledge hunting (using automatic extraction from text) module and an explicit reasoning module. Our end-to-end system built in such a manner improves on the accuracy of two of the available language model based approaches by 5.53% and 7.7% respectively. Overall our system achieves the state-of-the-art accuracy of 71.06% on the WSC dataset, an improvement of 7.36% over the previous best.

[1]  Chitta Baral,et al.  Identifying Various Kinds of Event Mentions in K-Parser Output , 2015, EVENTS@HLP-NAACL.

[2]  Xiaodong Liu,et al.  Multi-Task Deep Neural Networks for Natural Language Understanding , 2019, ACL.

[3]  Bhavana Dalvi,et al.  Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension , 2018, NAACL.

[4]  Heeyoung Lee,et al.  A Multi-Pass Sieve for Coreference Resolution , 2010, EMNLP.

[5]  Loizos Michael,et al.  Tackling the Winograd Schema Challenge Through Machine Logical Inferences , 2016, STAIRS.

[6]  Hector J. Levesque,et al.  The Winograd Schema Challenge , 2011, AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.

[7]  Jaime G. Carbonell,et al.  Anaphora Resolution: A Multi-Strategy Approach , 1988, COLING.

[8]  Chitta Baral,et al.  Towards Addressing the Winograd Schema Challenge - Building and Using a Semantic Parser and a Knowledge Hunting Module , 2015, IJCAI.

[9]  Lise Getoor,et al.  A short introduction to probabilistic soft logic , 2012, NIPS 2012.

[10]  Yongrui Chen,et al.  Convolutional Neural Network-Based Question Answering Over Knowledge Base with Type Constraint , 2018, CCKS.

[11]  Yuliya Lierler,et al.  The Winograd Schema Challenge and Reasoning about Correlation , 2015, AAAI Spring Symposia.

[12]  Peter Schüller,et al.  Tackling Winograd Schemas by Formalizing Relevance Theory in Knowledge Graphs , 2014, KR.

[13]  Yu Hu,et al.  Cause-Effect Knowledge Acquisition and Neural Association Model for Solving A Set of Winograd Schema Problems , 2017, IJCAI.

[14]  Jackie Chi Kit Cheung,et al.  A Knowledge Hunting Framework for Common Sense Reasoning , 2018, EMNLP.

[15]  Oren Etzioni,et al.  Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , 2018, ArXiv.

[16]  Jakob Uszkoreit,et al.  A Decomposable Attention Model for Natural Language Inference , 2016, EMNLP.

[17]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[18]  Quoc V. Le,et al.  A Simple Method for Commonsense Reasoning , 2018, ArXiv.

[19]  Vincent Ng,et al.  Machine Learning for Entity Coreference Resolution: A Retrospective Look at Two Decades of Research , 2017, AAAI.

[20]  Luke S. Zettlemoyer,et al.  Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language , 2015, EMNLP.

[21]  Jackie Chi Kit Cheung,et al.  A Generalized Knowledge Hunting Framework for the Winograd Schema Challenge , 2018, NAACL-HLT.

[22]  Peter Clark,et al.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , 2018, EMNLP.

[23]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.