The First Question Generation Shared Task Evaluation Challenge

The paper provides a detailed account of the First Shared Task Evaluation Challenge on Question Generation that took place in 2010. The campaign included two tasks that take text as input and produce text, i.e. questions, as output: Task A ‐ Question Generation from Paragraphs and Task B ‐ Question Generation from Sentences. Motivation, data sets, evaluation criteria, guidelines for judges, and results are presented for the two tasks. Lessons learned and advice for future Question Generation Shared Task Evaluation Challenges (QG-STEC) are also offered.

[1]  Le An Ha,et al.  A computer-aided environment for generating multiple-choice test items , 2006, Natural Language Engineering.

[2]  R. U S L A N M I T K O V,et al.  A computer-aided environment for generating multiple-choice test items , 2005 .

[3]  Arthur C. Graesser,et al.  Experiments on Generating Questions About Facts , 2009, CICLing.

[4]  Ellen M. Voorhees,et al.  The TREC-8 Question Answering Track Evaluation , 2000, TREC.

[5]  Jack Mostow,et al.  Generating Instruction Automatically for the Reading Strategy of Self-Questioning , 2009, AIED.

[6]  Arthur C. Graesser,et al.  Questions and information systems , 1992 .

[7]  H WolfeJohn.  Automatic question generation from text - an aid to independent study , 1976 .

[8]  Paul Piwek,et al.  Generating Questions from OpenLearn study units , 2009 .

[9]  Rashmi Prasad,et al.  Question Generation from Paragraphs at UPenn: QGSTEC System Description , 2010 .

[10]  Andrea Varga,et al.  WLV: A Question Generation System for the QGSTEC 2010 Task B , 2010 .

[11]  Albert Gatt,et al.  The GREC Challenge 2008: Overview and Evaluation Results , 2008, INLG.

[12]  Robert Dale,et al.  Building applied natural language generation systems , 1997, Natural Language Engineering.

[13]  Anja Belz,et al.  Comparing Rating Scales and Preference Judgements in Language Evaluation , 2010, INLG.

[14]  Helmut Prendinger,et al.  A Novel Discourse Parser Based on Support Vector Machine Classification , 2009, ACL.

[15]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[16]  Anja Belz,et al.  The GREC Challenges 2010: Overview and Evaluation Results , 2010, INLG.

[17]  Kristy Elizabeth Boyer,et al.  Proceedings of QG2010: The Third Workshop on Question Generation , 2010 .

[18]  Albert Gatt,et al.  The TUNA-REG Challenge 2009: Overview and Evaluation Results , 2009, ENLG.

[19]  A. Collins,et al.  Cognition and learning. , 1996 .

[20]  Marilyn A. Walker,et al.  Training a sentence planner for spoken dialogue using boosting , 2002, Comput. Speech Lang..

[21]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[22]  Xuchen Yao,et al.  Question Generation with Minimal Recursion Semantics , 2010 .

[23]  John S. White,et al.  Review of Questions and information systems by Thomas W. Lauer, Eileen Peacock, and Arthur C. Graesser. Lawrence Erlbaum Associates 1992. , 1993 .

[24]  John H. Wolfe,et al.  Automatic question generation from text - an aid to independent study , 1976, SIGCSE '76.

[25]  Johanna D. Moore,et al.  Report on the Second NLG Challenge on Generating Instructions in Virtual Environments (GIVE-2) , 2010, INLG.

[26]  Jack Mostow,et al.  Can Automated Questions Scaffold Children's Reading Comprehension? , 2004, Intelligent Tutoring Systems.

[27]  Mitsuru Ishizuka,et al.  T2D: Generating Dialogues Between Virtual Agents Automatically from Text , 2007, IVA.

[28]  Jack Mostow,et al.  Using Automated Questions to Assess Reading Comprehension, Vocabulary, and Effects of Tutorial Interventions , 2004 .

[29]  Chin-Yew Lin,et al.  Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics , 2004, ACL.

[30]  Tsukasa Hirashima,et al.  Automated Question Generation Methods for Intelligent English Learning Systems and its Evaluation , 2001 .

[31]  Paul Piwek,et al.  Data-oriented Monologue-to-Dialogue Generation , 2011, ACL.

[32]  Sadid A. Hasan,et al.  Automation of Question Generation From Sentences , 2011 .