Leaf: Multiple-Choice Question Generation

Testing with quiz questions has proven to be an effective way to assess and improve the educational process. However, manually creating quizzes is tedious and time-consuming. To address this challenge, we present Leaf, a system for generating multiple-choice questions from factual text. In addition to being very well suited for the classroom, Leaf could also be used in an industrial setting, e.g., to facilitate onboarding and knowledge sharing, or as a component of chatbots, question answering systems, or Massive Open Online Courses (MOOCs). The code and the demo are available on GitHub.

[1]  John Liu,et al.  sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings , 2015, ArXiv.

[2]  Ming Zhou,et al.  Neural Question Generation from Text: A Preliminary Study , 2017, NLPCC.

[3]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[4]  Oren Etzioni,et al.  Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , 2018, ArXiv.

[5]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[6]  Hao Tian,et al.  ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation , 2020, ArXiv.

[7]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[8]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[9]  Philip Bachman,et al.  NewsQA: A Machine Comprehension Dataset , 2016, Rep4NLP@ACL.

[10]  Oren Etzioni,et al.  From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project , 2020, AI Mag..

[11]  Suzan Verberne,et al.  Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering , 2020, ArXiv.

[12]  Hiroshi Nakagawa,et al.  WebExperimenter for Multiple-Choice Question Generation , 2005, HLT/EMNLP.

[13]  Ming Zhou,et al.  Question Generation for Question Answering , 2017, EMNLP.

[14]  Peter Clark,et al.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , 2018, EMNLP.

[15]  Preslav Nakov,et al.  Beyond English-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for Bulgarian , 2019, RANLP.

[16]  Preslav Nakov,et al.  EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering , 2020, EMNLP.

[17]  Peter Clark,et al.  QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationships , 2018, AAAI.

[18]  Jianfeng Gao,et al.  UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training , 2020, ICML.

[19]  Sebastian Riedel,et al.  MLQA: Evaluating Cross-lingual Extractive Question Answering , 2019, ACL.

[20]  Henry L. Roediger,et al.  Ten benefits of testing and their applications to educational practice. , 2011 .

[21]  Piji Li,et al.  Generating Distractors for Reading Comprehension Questions from Real Examinations , 2018, AAAI.

[22]  Yao-Chung Fan,et al.  A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies. , 2020, FINDINGS.

[23]  Graham Neubig,et al.  XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization , 2020, ICML.

[24]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[25]  Alon Lavie,et al.  METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments , 2007, WMT@ACL.

[26]  Bijan Parsia,et al.  A Systematic Review of Automatic Question Generation for Educational Purposes , 2019, International Journal of Artificial Intelligence in Education.

[27]  John Lee,et al.  Restatement and Question Generation for Counsellor Chatbot , 2021, NLP4POSIMPACT.

[28]  Xiaodong Liu,et al.  Unified Language Model Pre-training for Natural Language Understanding and Generation , 2019, NeurIPS.

[29]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[30]  R. Mitkov,et al.  Computer-Aided Generation of Multiple-Choice Tests , 2003, International Conference on Natural Language Processing and Knowledge Engineering, 2003. Proceedings. 2003.

[31]  Kiet Van Nguyen,et al.  Enhancing Lexical-Based Approach With External Knowledge for Vietnamese Multiple-Choice Machine Reading Comprehension , 2020, IEEE Access.

[32]  Deyi Xiong,et al.  BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels , 2019, EMNLP.

[33]  Eunsol Choi,et al.  TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages , 2020, Transactions of the Association for Computational Linguistics.

[34]  Xinya Du,et al.  Learning to Ask: Neural Question Generation for Reading Comprehension , 2017, ACL.

[35]  Melissa Roemmele,et al.  AnswerQuest: A System for Generating Question-Answer Items from Multi-Paragraph Documents , 2021, EACL.

[36]  Takenobu Tokunaga,et al.  Evaluation of automatically generated English vocabulary questions , 2017, Research and Practice in Technology Enhanced Learning.

[37]  Paul Piwek,et al.  Evaluation methodologies in Automatic Question Generation 2013-2018 , 2018, INLG.

[38]  Yue Zhang,et al.  Leveraging Context Information for Natural Question Generation , 2018, NAACL.

[39]  Ming Zhou,et al.  ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training , 2020, FINDINGS.

[40]  Mohit Iyyer,et al.  Generating Question-Answer Hierarchies , 2019, ACL.

[41]  Preslav Nakov,et al.  Generating Answer Candidates for Quizzes and Answer-Aware Question Generators , 2021, RANLP.

[42]  Guokun Lai,et al.  RACE: Large-scale ReAding Comprehension Dataset From Examinations , 2017, EMNLP.

[43]  Kevin Gimpel,et al.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations , 2019, ICLR.

[44]  Omer Levy,et al.  RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019, ArXiv.

[45]  Zornitsa Kozareva,et al.  Few-shot Learning with Multilingual Language Models , 2021, ArXiv.