Good Question! Statistical Ranking for Question Generation

We address the challenge of automatically generating questions from reading materials for educational practice and assessment. Our approach is to overgenerate questions, then rank them. We use manually written rules to perform a sequence of general purpose syntactic transformations (e.g., subject-auxiliary inversion) to turn declarative sentences into questions. These questions are then ranked by a logistic regression model trained on a small, tailored dataset consisting of labeled output from our system. Experimental results show that ranking nearly doubles the percentage of questions rated as acceptable by annotators, from 27% of all questions to 52% of the top ranked 20% of questions.

[1]  Ido Dagan,et al.  A Probabilistic Classification Approach for Lexical Textual Entailment , 2005, AAAI.

[2]  Vasile Rus,et al.  The 2nd Workshop on Question Generation , 2009, AIED.

[3]  Kevin Knight,et al.  Generation that Exploits Corpus-Based Statistical Knowledge , 1998, ACL.

[4]  Eduard H. Hovy,et al.  The Use of External Knowledge of Factoid QA , 2001, TREC.

[5]  Michael Collins,et al.  Head-Driven Statistical Models for Natural Language Parsing , 2003, CL.

[6]  Richard M. Schwartz,et al.  Hedge Trimmer: A Parse-and-Trim Approach to Headline Generation , 2003, HLT-NAACL 2003.

[7]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[8]  Robert Dale,et al.  Building applied natural language generation systems , 1997, Natural Language Engineering.

[9]  John Cocke,et al.  A Statistical Approach to Machine Translation , 1990, CL.

[10]  Roger Levy,et al.  Tregex and Tsurgeon: tools for querying and manipulating tree data structures , 2006, LREC.

[11]  S. Cessie,et al.  Ridge Estimators in Logistic Regression , 1992 .

[12]  Richard M. Schwartz,et al.  An Algorithm that Learns What's in a Name , 1999, Machine Learning.

[13]  R. U S L A N M I T K O V,et al.  A computer-aided environment for generating multiple-choice test items , 2005 .

[14]  Noah A. Smith,et al.  Question Generation via Overgenerating Transformations and Ranking , 2009 .

[15]  Dan Klein,et al.  Fast Exact Inference with a Factored Model for Natural Language Parsing , 2002, NIPS.

[16]  Tsukasa Hirashima,et al.  Automated Question Generation Methods for Intelligent English Learning Systems and its Evaluation , 2001 .

[17]  Marilyn A. Walker,et al.  SPoT: A Trainable Sentence Planner , 2001, NAACL.

[18]  Donna M Gates,et al.  Automatically Generating Reading Comprehension Look-Back Strategy: Questions from Expository Texts , 2008 .

[19]  Noam Chomsky,et al.  Conditions on transformations , 1971 .

[20]  R. Mitkov,et al.  Computer-Aided Generation of Multiple-Choice Tests , 2003, International Conference on Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003.

[21]  Sanda M. Harabagiu,et al.  Experiments with Interactive Question-Answering , 2005, ACL.

[22]  Daniel Marcu,et al.  Statistics-Based Summarization - Step One: Sentence Compression , 2000, AAAI/IAAI.

[23]  Michael Collins,et al.  Discriminative Reranking for Natural Language Parsing , 2000, CL.

[24]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Novelty Track. , 2005 .

[25]  Jimmy J. Lin,et al.  Overview of the TREC 2007 Question Answering Track , 2008, TREC.

[26]  Nancy Ide,et al.  The American National Corpus First Release , 2004, LREC.

[27]  Le An Ha,et al.  A computer-aided environment for generating multiple-choice test items , 2006, Natural Language Engineering.

[28]  Michael Gamon,et al.  The PYTHY Summarization System: Microsoft Research at DUC 2007 , 2007 .

[29]  Hermann Ney,et al.  Improved backing-off for M-gram language modeling , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.

[30]  Chris Brockett,et al.  Automatically Constructing a Corpus of Sentential Paraphrases , 2005, IJCNLP.

[31]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[32]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[33]  Volume Assp,et al.  ACOUSTICS. SPEECH. AND SIGNAL PROCESSING , 1983 .

[34]  John Robert Ross,et al.  Constraints on variables in syntax , 1967 .

[35]  A. Goldberg.  Constructions at Work: The Nature of Generalization in Language , 2006 .

[36]  Thomas P. Minka,et al.  Gates , 2008, NIPS.

[37]  Eduard H. Hovy,et al.  Learning surface text patterns for a Question Answering System , 2002, ACL.

[38]  Stephen R. Anderson,et al.  A Festschrift for Morris Halle , 1973 .

[39]  Daniel Marcu,et al.  A Noisy-Channel Approach to Question Answering , 2003, ACL.