Controllable Open-ended Question Generation with A New Question Type Ontology

We investigate the less-explored task of generating open-ended questions that are typically answered by multiple sentences. We first define a new question type ontology which differentiates the nuanced nature of questions better than widely used question words. A new dataset with 4, 959 questions is labeled based on the new ontology. We then propose a novel question type-aware question generation framework, augmented by a semantic graph representation, to jointly predict question focuses and produce the question. Based on this framework, we further use both exemplars and automatically generated templates to improve controllability and diversity. Experiments on two newly collected large-scale datasets show that our model improves question quality over competitive comparisons based on automatic metrics. Human judges also rate our model outputs highly in answerability, coverage of scope, and overall quality. Finally, our model variants with templates can produce questions with enhanced controllability and diversity.

[1]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[2]  Ramón Fernández Astudillo,et al.  On the Importance of Diversity in Question Generation for QA , 2020, ACL.

[3]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[4]  Yanjun Ma,et al.  Answer-focused and Position-aware Neural Question Generation , 2018, EMNLP.

[5]  Mohit Bansal,et al.  Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering , 2019, EMNLP.

[6]  Patricia Shapley Online Education to Develop Complex Reasoning Skills in Organic Chemistry , 2019, Online Learning.

[7]  Arthur C. Graesser,et al.  Question Generation from Concept Maps , 2012, Dialogue Discourse.

[8]  Lidong Bing,et al.  Improving Question Generation With to the Point Context , 2019, EMNLP.

[9]  Jie Zhang,et al.  Diversify Question Generation with Continuous Content Selectors and Question Type Modeling , 2020, FINDINGS.

[10]  Wenjie Zhou,et al.  Multi-Task Learning with Language Modeling for Question Generation , 2019, EMNLP.

[11]  Catherine Havasi,et al.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge , 2016, AAAI.

[12]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[13]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[14]  Mitesh M. Khapra,et al.  Towards a Better Metric for Evaluating Question Generation Systems , 2018, EMNLP.

[15]  Wenjie Zhou,et al.  Question-type Driven Question Generation , 2019, EMNLP.

[16]  R. Mitkov,et al.  Computer-Aided Generation of Multiple-Choice Tests , 2003, International Conference on Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003.

[17]  Myle Ott,et al.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling , 2019, NAACL.

[18]  Rashmi Prasad,et al.  Question Generation from Paragraphs at UPenn: QGSTEC System Description , 2010 .

[19]  Jimmy J. Lin,et al.  Simple BERT Models for Relation Extraction and Semantic Role Labeling , 2019, ArXiv.

[20]  Yoshua Bengio,et al.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , 2018, EMNLP.

[21]  Harry Shum,et al.  From Eliza to XiaoIce: challenges and opportunities with social chatbots , 2018, Frontiers of Information Technology & Electronic Engineering.

[22]  Yan Xu,et al.  Multi-hop Question Generation with Graph Convolutional Network , 2020, FINDINGS.

[23]  Samuel A. Livingston Constructed-Response Test Questions: Why We Use Them; How We Score Them. R&D Connections. Number 11. , 2009 .

[24]  Hal Daumé,et al.  Answer-based Adversarial Training for Generating Clarification Questions , 2019, NAACL.

[25]  Yansong Feng,et al.  Semantic Graphs for Generating Deep Questions , 2020, ACL.

[26]  Kenneth Tobin,et al.  Research on Science Laboratory Activities: In Pursuit of Better Questions and Answers to Improve Learning , 1990 .

[27]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[28]  Di Niu,et al.  Asking Questions the Human Way: Scalable Question-Answer Generation from Text Corpus , 2020, WWW.

[29]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[30]  Jennifer L Skeem,et al.  How do , 2012 .

[31]  Hal Daumé,et al.  Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information , 2018, ACL.