Promptiverse: Scalable Generation of Scaffolding Prompts Through Human-AI Hybrid Knowledge Graph Annotation

Online learners are hugely diverse with varying prior knowledge, but most instructional videos online are created to be one-size-fits-all. Thus, learners may struggle to understand the content by only watching the videos. Providing scaffolding prompts can help learners overcome these struggles through questions and hints that relate different concepts in the videos and elicit meaningful learning. However, serving diverse learners would require a spectrum of scaffolding prompts, which incurs high authoring effort. In this work, we introduce Promptiverse, an approach for generating diverse, multi-turn scaffolding prompts at scale, powered by numerous traversal paths over knowledge graphs. To facilitate the construction of the knowledge graphs, we propose a hybrid human-AI annotation tool, Grannotate. In our study (N=24), participants produced 40 times more on-par quality prompts with higher diversity, through Promptiverse and Grannotate, compared to hand-designed prompts. Promptiverse presents a model for creating diverse and adaptive learning experiences online.

[1]  Arnab Bhattacharya,et al.  Sangrahaka: a tool for annotating and querying knowledge graphs , 2021, ESEC/SIGSOFT FSE.

[2]  Brian Lester,et al.  The Power of Scale for Parameter-Efficient Prompt Tuning , 2021, EMNLP.

[3]  Stella Biderman,et al.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow , 2021 .

[4]  Zhengxiao Du,et al.  GPT Understands, Too , 2021, AI Open.

[5]  Brian Y. Lim,et al.  Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation , 2021, CHI.

[6]  Sameer Singh,et al.  Eliciting Knowledge from Language Models Using Automatically Generated Prompts , 2020, EMNLP.

[7]  Dawn Song,et al.  Language Models are Open Knowledge Graphs , 2020, ArXiv.

[8]  Eric Horvitz,et al.  Extracting a Knowledge Base of Mechanisms from COVID-19 Papers , 2020, NAACL.

[9]  Iryna Gurevych,et al.  From Zero to Hero: Human-In-The-Loop Entity Linking in Low Resource Domains , 2020, ACL.

[10]  Alan Ritter,et al.  Fluent Response Generation for Conversational Question Answering , 2020, ACL.

[11]  Eytan Adar,et al.  texSketch: Active Diagramming through Pen-and-Ink Annotations , 2020, CHI.

[12]  Sebastian Hobert,et al.  Sara, the Lecturer: Improving Learning in Online Education with a Scaffolding-Based Conversational Agent , 2020, CHI.

[13]  Nicholas Jing Yuan,et al.  Integrating Graph Contextualized Knowledge into Pre-trained Language Models , 2019, FINDINGS.

[14]  Hannaneh Hajishirzi,et al.  Entity, Relation, and Event Extraction with Contextualized Span Representations , 2019, EMNLP.

[15]  Deng Cai,et al.  Reinforced Dynamic Reasoning for Conversational Question Generation , 2019, ACL.

[16]  Maosong Sun,et al.  ERNIE: Enhanced Language Representation with Informative Entities , 2019, ACL.

[17]  James A. Landay,et al.  QuizBot: A Dialogue-based Adaptive Learning System for Factual Knowledge , 2019, CHI.

[18]  Neil Mercer,et al.  Scaffolding Student Understanding in Small-Group Work: Students’ Uptake of Teacher Support in Subsequent Small-Group Interaction , 2018, Journal of the Learning Sciences.

[19]  Mari Ostendorf,et al.  Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction , 2018, EMNLP.

[20]  Danqi Chen,et al.  CoQA: A Conversational Question Answering Challenge , 2018, TACL.

[21]  Eunsol Choi,et al.  QuAC: Question Answering in Context , 2018, EMNLP.

[22]  Maria-Esther Vidal,et al.  Towards a Knowledge Graph for Science , 2018, WIMS.

[23]  Iryna Gurevych,et al.  The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation , 2018, COLING.

[24]  Behrang Q. Zadeh,et al.  SemEval-2018 Task 7: Semantic Relation Extraction and Classification in Scientific Papers , 2018, *SEMEVAL.

[25]  Joseph Jay Williams,et al.  Understanding the Effect of In-Video Prompting on Learners and Instructors , 2018, CHI.

[26]  Juho Kim,et al.  ConceptScape: Collaborative Concept Mapping for Video Learning , 2018, CHI.

[27]  Krzysztof Z. Gajos,et al.  Semantically Far Inspirations Considered Harmful?: Accounting for Cognitive States in Collaborative Ideation , 2017, Creativity & Cognition.

[28]  Geza Kovacs,et al.  Effects of In-Video Quizzes on MOOC Lecture Viewing , 2016, L@S.

[29]  Leonidas J. Guibas,et al.  Deep Knowledge Tracing , 2015, NIPS.

[30]  Ömer Faruk Vural,et al.  The Impact of a Question-Embedded Video-based Learning Tool on E-learning * , 2013 .

[31]  Jeffrey D. Karpicke,et al.  Retrieval-Based Learning: A Perspective for Enhancing Meaningful Learning , 2012 .

[32]  Shuying Shen,et al.  A Prototype Tool Set to Support Machine-Assisted Annotation , 2012, BioNLP@HLT-NAACL.

[33]  Sampo Pyysalo,et al.  brat: a Web-based Tool for NLP-Assisted Text Annotation , 2012, EACL.

[34]  Preslav Nakov,et al.  SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals , 2009, SEW@NAACL-HLT.

[35]  David S. Kreiner,et al.  Incorporating Active Learning with PowerPoint-Based Lectures Using Content-Based Questions , 2009 .

[36]  Preslav Nakov,et al.  SemEval-2007 Task 04: Classification of Semantic Relations between Nominals , 2007, Fourth International Workshop on Semantic Evaluations (SemEval-2007).

[37]  R. Mayer,et al.  Interactive Multimodal Learning Environments , 2007 .

[38]  S. Ainsworth DeFT: A Conceptual Framework for Considering Learning with Multiple Representations. , 2006 .

[39]  Timothy J. Lawson,et al.  Guiding Questions Enhance Student Learning from Educational Videos , 2006 .

[40]  Joseph D. Novak,et al.  Meaningful learning: The essential factor for conceptual change in limited or inappropriate propositional hierarchies leading to empowerment of learners , 2002 .

[41]  Angela M. O'Donnell,et al.  Knowledge Maps as Scaffolds for Cognitive Processing , 2002 .

[42]  Takashi Yamauchi,et al.  Learning from human tutoring , 2001, Cogn. Sci..

[43]  David P. Ausubel,et al.  The Acquisition and Retention of Knowledge: A Cognitive View , 2000 .

[44]  Daniel H. Robinson,et al.  Visual argument: Graphic organizers are superior to outlines in improving learning from text. , 1995 .

[45]  P. Rousseeuw Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[46]  William C. Mann,et al.  RHETORICAL STRUCTURE THEORY: A THEORY OF TEXT ORGANIZATION , 1987 .

[47]  David G. Ausubel Cognitive Structure and the Facilitation of Meaningful Verbal Learning1 , 1963 .

[48]  D. Ausubel,et al.  A subsumption theory of meaningful verbal learning and retention. , 1962, The Journal of general psychology.

[49]  Percy Liang,et al.  Prefix-Tuning: Optimizing Continuous Prompts for Generation , 2021, ACL.

[50]  Sharad Goel,et al.  MathBot: A Personalized Conversational Agent for Learning Math , 2020 .

[51]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[52]  Alastair R. Beresford,et al.  Investigating Engagement with In-Video Quiz Questions in a Programming Course , 2016, IEEE Transactions on Learning Technologies.

[53]  Shuying Shen,et al.  BoB, a best-of-breed automated text de-identification system for VHA clinical documents , 2013, J. Am. Medical Informatics Assoc..

[54]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[55]  Joel A. Michael,et al.  Hinting as a Tactic in One-on-One Tutoring , 1996 .

[56]  Michelene T. H. Chi,et al.  Eliciting Self-Explanations Improves Understanding , 1994, Cogn. Sci..