MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

We introduce a large dataset of narrative texts and questions about these texts, intended to be used in a machine comprehension task that requires reasoning using commonsense knowledge. Our dataset complements similar datasets in that we focus on stories about everyday activities, such as going to the movies or working in the garden, and that the questions require commonsense knowledge, or more specifically, script knowledge, to be answered. We show that our mode of data collection via crowdsourcing results in a substantial amount of such inference questions. The dataset forms the basis of a shared task on commonsense and script knowledge organized at SemEval 2018 and provides challenging test cases for the broader natural language understanding community.

[1]  Guokun Lai,et al.  RACE: Large-scale ReAding Comprehension Dataset From Examinations , 2017, EMNLP.

[2]  Raymond J. Mooney,et al.  Using Sentence-Level LSTM Language Models for Script Inference , 2016, ACL.

[3]  Nathanael Chambers,et al.  Unsupervised Learning of Narrative Event Chains , 2008, ACL.

[4]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[5]  Simon Ostermann,et al.  InScript: Narrative texts annotated with script information , 2016, LREC.

[6]  Nathanael Chambers,et al.  Unsupervised Learning of Narrative Schemas and their Participants , 2009, ACL.

[7]  Erik T. Mueller,et al.  Open Mind Common Sense: Knowledge Acquisition from the General Public , 2002, OTM.

[8]  Eunsol Choi,et al.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , 2017, ACL.

[9]  Jeffrey Pennington,et al.  GloVe: Global Vectors for Word Representation , 2014, EMNLP.

[10]  Ivan Titov,et al.  Learning Semantic Script Knowledge with Event Embeddings , 2014, ICLR.

[11]  Raymond J. Mooney,et al.  Statistical Script Learning with Multi-Argument Events , 2014, EACL.

[12]  Stefan Thater,et al.  Inducing Script Structure from Crowdsourced Event Descriptions via Semi-Supervised Clustering , 2017, LSDSem@EACL.

[13]  Philip Bachman,et al.  NewsQA: A Machine Comprehension Dataset , 2016, Rep4NLP@ACL.

[14]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[15]  Jason Weston,et al.  The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations , 2015, ICLR.

[16]  Danqi Chen,et al.  A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task , 2016, ACL.

[17]  Phil Blunsom,et al.  Teaching Machines to Read and Comprehend , 2015, NIPS.

[18]  Matthew Richardson,et al.  MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text , 2013, EMNLP.

[19]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[20]  Ivan Titov,et al.  Inducing Neural Models of Script Knowledge , 2014, CoNLL.

[21]  Ashutosh Modi,et al.  Event Embeddings for Semantic Script Modeling , 2016, CoNLL.

[22]  Stefan Thater,et al.  A Crowdsourced Database of Event Sequence Descriptions for the Acquisition of High-quality Script Knowledge , 2016, LREC.

[23]  Manfred Pinkal,et al.  Learning Script Knowledge with Web Experiments , 2010, ACL.