Few-shot Question Generation for Personalized Feedback in Intelligent Tutoring Systems

Existing work on generating hints in Intelligent Tutoring Systems (ITS) focuses mostly on manual and non-personalized feedback. In this work, we explore automatically generated questions as personalized feedback in an ITS. Our personalized feedback can pinpoint correct and incorrect or missing phrases in student answers as well as guide them towards correct answer by asking a question in natural language. Our approach combines cause-effect analysis to break down student answers using text similarity-based NLP Transformer models to identify correct and incorrect or missing parts. We train a few-shot Neural Question Generation and Question Re-ranking models to show questions addressing components missing in the student answers which steers students towards the correct answer. Our model vastly outperforms both simple and strong baselines in terms of student learning gains by 45% and 23% respectively when tested in a real dialogue-based ITS. Finally, we show that our personalized corrective feedback system has the potential to improve Generative Question Answering systems.

[1]  Dung D. Vu,et al.  A New Era: Intelligent Tutoring Systems Will Transform Online Learning for Millions , 2022, ArXiv.

[2]  Joelle Pineau,et al.  Automated Data-Driven Generation of Personalized Pedagogical Interventions in Intelligent Tutoring Systems , 2021, International Journal of Artificial Intelligence in Education.

[3]  Megha Srivastava,et al.  Question Generation for Adaptive Education , 2021, ACL.

[4]  Massimo Piccardi,et al.  BERTTune: Fine-Tuning Neural Machine Translation with BERTScore , 2021, ACL.

[5]  Gordon Fraser,et al.  Guiding Next-Step Hint Generation Using Automated Tests , 2021, ITiCSE.

[6]  Siva Reddy,et al.  Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrieval , 2021, EMNLP.

[7]  Dung D. Vu,et al.  Comparative Study of Learning Outcomes for Online Learning Platforms , 2021, AIED.

[8]  Ekaterina Kochmar,et al.  Deep Discourse Analysis for Generating Personalized Feedback in Intelligent Tutor Systems , 2021, AAAI.

[9]  Eunsol Choi,et al.  QED: A Framework and Dataset for Explanations in Question Answering , 2020, Transactions of the Association for Computational Linguistics.

[10]  Wei Liu,et al.  Low-Resource Generation of Multi-hop Reasoning Questions , 2020, ACL.

[11]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[12]  Tassilo Klein,et al.  Learning to Answer by Learning to Ask: Getting the Best of GPT-2 and BERT Worlds , 2019, ArXiv.

[13]  J. Weston,et al.  Adversarial NLI: A New Benchmark for Natural Language Understanding , 2019, ACL.

[14]  Sung-Hyon Myaeng,et al.  Let Me Know What to Ask: Interrogative-Word-Aware Question Generation , 2019, EMNLP.

[15]  Omer Levy,et al.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension , 2019, ACL.

[16]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[17]  Irena Koprinska,et al.  A Survey of Automated Programming Hint Generation: The HINTS Framework , 2019, ACM Comput. Surv..

[18]  Iryna Gurevych,et al.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks , 2019, EMNLP.

[19]  William Yang Wang,et al.  Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarization , 2019, EMNLP.

[20]  Xiaodong Liu,et al.  Unified Language Model Pre-training for Natural Language Understanding and Generation , 2019, NeurIPS.

[21]  Kilian Q. Weinberger,et al.  BERTScore: Evaluating Text Generation with BERT , 2019, ICLR.

[22]  Manaal Faruqui,et al.  Identifying Well-formed Natural Language Questions , 2018, EMNLP.

[23]  Ming Liu,et al.  Automated Essay Feedback Generation and Its Impact on Revision , 2017, IEEE Transactions on Learning Technologies.

[24]  Xinya Du,et al.  Learning to Ask: Neural Question Generation for Reading Comprehension , 2017, ACL.

[25]  Markus Freitag,et al.  Beam Search Strategies for Neural Machine Translation , 2017, NMT@ACL.

[26]  Hai Zhuge,et al.  The Role of Cause-Effect Link within Scientific Paper , 2016, 2016 12th International Conference on Semantics, Knowledge and Grids (SKG).

[27]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Michael Eagle,et al.  Experimental Evaluation of Automatic Hint Generation for a Logic Tutor , 2011, Int. J. Artif. Intell. Educ..

[30]  Jörg H. Siekmann,et al.  Natural Language Dialog with a Tutor System for Mathematical Proofs , 2005, Cognitive Systems.

[31]  Paul Blayney,et al.  Automated formative feedback and summative assessment using individualised spreadsheet assignments , 2004 .

[32]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[33]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[34]  Judith D. Wilson,et al.  Artificial Intelligence and Tutoring Systems , 1990 .

[35]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[36]  Shashi Narayan,et al.  A Thorough Evaluation of Task-Specific Pretraining for Summarization , 2021, EMNLP.

[37]  Jianzhong Qi,et al.  Latent Reasoning for Low-Resource Question Generation , 2021, FINDINGS.

[38]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[39]  Yao Zhao,et al.  Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks , 2018, EMNLP.

[40]  Jie Yang,et al.  Delft University of Technology LearningQ: A Large-scale Dataset for Educational Question Generation , 2018 .

[41]  Kurt VanLehn,et al.  Representation and Reasoning for Deeper Natural Language Understanding in a Physics Tutoring System , 2006, FLAIRS.