An Evaluation of Pedagogical Tutorial Tactics for a Natural Language Tutoring System: A Reinforcement Learning Approach

Pedagogical strategies are policies for a tutor to decide the next action when there are multiple actions available. When the content is controlled to be the same across experimental conditions, there has been little evidence that tutorial decisions have an impact on students' learning. In this paper, we applied Reinforcement Learning (RL) to induce two sets of pedagogical policies from pre-existing human interaction data. The NormGain set was derived with the goal of enhancing tutorial decisions that contribute to learning while the InvNormGain set was derived with the goal of enhancing those decisions that contribute less or even nothing to learning. The two sets were then tested with human students. Our results show that when the content was controlled to be the same, different pedagogical policies did make a difference in learning and more specifically, the NormGain students outperformed their peers. Overall our results suggest that content exposure and practice opportunities can help students to learn even when tutors have poor pedagogical tutorial tactics. However, with effective tutorial tactics, students can learn even more.

[1]  Paul J. Feltovich,et al.  Categorization and Representation of Physics Problems by Experts and Novices , 1981, Cogn. Sci..

[2]  Johanna D. Moore,et al.  A comparative evaluation of socratic versus didactic tutoring , 2001 .

[3]  S. Young,et al.  Scaling POMDPs for Spoken Dialog Management , 2007, IEEE Transactions on Audio, Speech, and Language Processing.

[4]  Sandra Katz,et al.  An Approach to Analyzing the Role and Structure of Reflective Dialogue , 1999 .

[5]  Andrew G. Barto,et al.  Reinforcement learning , 1998 .

[6]  Peter Johnson,et al.  Different explanatory dialogue styles and their effects on knowledge acquisition by novices , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[7]  Paloma Martínez,et al.  Err is Human: Building a safer health system , 2003 .

[8]  K. VanLehn,et al.  Why Do Only Some Events Cause Learning During Human Tutoring? , 2003 .

[9]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[10]  K. Koedinger,et al.  Exploring the Assistance Dilemma in Experiments with Cognitive Tutors , 2007 .

[11]  Susan E. Newman,et al.  Cognitive Apprenticeship: Teaching the Craft of Reading, Writing, and Mathematics. Technical Report No. 403. , 1987 .

[12]  Ralph T. Putnam Structuring and Adjusting Content for Students: A Study of Live and Simulated Tutoring of Addition , 1987 .

[13]  M. Chi,et al.  The Content of Physics Self-Explanations , 1991 .

[14]  Paloma Martínez,et al.  Learning teaching strategies in an Adaptive and Intelligent Educational System through Reinforcement Learning , 2009, Applied Intelligence.

[15]  Kurt VanLehn,et al.  Do Micro-Level Tutorial Decisions Matter: Applying Reinforcement Learning to Induce Pedagogical Tutorial Tactics , 2010, Intelligent Tutoring Systems.

[16]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[17]  Joel R. Tetreault,et al.  Using Reinforcement Learning to Build a Better Model of Dialogue State , 2006, EACL.

[18]  Kurt VanLehn,et al.  The Andes Physics Tutoring System: Lessons Learned , 2005, Int. J. Artif. Intell. Educ..

[19]  J. Gregory Trafton,et al.  Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems , 1992 .

[20]  Cristina Conati,et al.  Toward Computer-Based Support of Meta-Cognitive Skills: a Computational Framework to Coach Self-Explanation , 2000 .

[21]  Joel R. Tetreault,et al.  A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems , 2008, Speech Commun..

[22]  Vincent Aleven,et al.  Evaluating the Effectiveness of a Tutorial Dialogue System for Self-Explanation , 2004, Intelligent Tutoring Systems.

[23]  Oliver Lemon,et al.  User simulations for online adaptation and knowledge-alignment in troubleshooting dialogue systems , 2008 .

[24]  Joseph P. Magliano,et al.  Collaborative dialogue patterns in naturalistic one-to-one tutoring , 1995 .

[25]  Ivon Arroyo,et al.  AgentX: Using Reinforcement Learning to Improve the Effectiveness of Intelligent Tutoring Systems , 2004, Intelligent Tutoring Systems.

[26]  Kenneth R. Koedinger,et al.  Is Self-Explanation Always Better? The Effects of Adding Self-Explanation Prompts to an English Grammar Tutor , 2009 .

[27]  Allan Collins,et al.  Cognitive Apprenticeship: Teaching the Craft of Reading, Writing, and Mathematics. Technical Report No. 403. , 1987 .

[28]  B. Bloom The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring , 1984 .

[29]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[30]  Kurt VanLehn,et al.  To Elicit Or To Tell: Does It Matter? , 2009, AIED.

[31]  Kurt VanLehn The Interaction Plateau: Answer-Based Tutoring < Step-Based Tutoring = Natural Tutoring , 2008, Intelligent Tutoring Systems.

[32]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[33]  L. Vygotsky Interaction between learning and development , 1978 .

[34]  Joel R. Tetreault,et al.  Estimating the Reliability of MDP Policies: a Confidence Interval Approach , 2007, HLT-NAACL.

[35]  Michelene T. H. Chi,et al.  Observing Tutorial Dialogues Collaboratively: Insights About Human Tutoring Effectiveness From Vicarious Learning , 2008, Cogn. Sci..

[36]  Arthur C. Graesser,et al.  Intelligent Tutoring Systems with Conversational Dialogue , 2001, AI Mag..

[37]  Sandra Katz,et al.  Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes , 2007, AIED.

[38]  L. Resnick,et al.  Knowing, Learning, and Instruction , 2018 .

[39]  F. Reif,et al.  Teaching scientific thinking skills: Students and computers coaching each other , 1999 .

[40]  Claus Zinn,et al.  Generating Tutorial Feedback with Affect , 2004, FLAIRS.

[41]  Sidney K. D'Mello,et al.  Dialogue Modes in Expert Tutoring , 2008, Intelligent Tutoring Systems.

[42]  Tom Routen,et al.  Intelligent Tutoring Systems , 1996, Lecture Notes in Computer Science.

[43]  Ibrahim A. Halloun,et al.  The initial knowledge state of college physics students , 1985 .

[44]  Dorothea P. Simon,et al.  Expert and Novice Performance in Solving Physics Problems , 1980, Science.

[45]  Niels Ole Bernsen,et al.  Designing Interactive Speech Systems , 1998, Springer London.

[46]  Marilyn A. Walker,et al.  Reinforcement Learning for Spoken Dialogue Systems , 1999, NIPS.

[47]  Joseph E. Beck,et al.  ADVISOR: A Machine Learning Architecture for Intelligent Tutor Construction , 2000, AAAI/IAAI.

[48]  Paloma Martínez,et al.  Reinforcement learning of pedagogical policies in adaptive and intelligent educational systems , 2009, Knowl. Based Syst..

[49]  Richard Reviewer-Granger Unified Theories of Cognition , 1991, Journal of Cognitive Neuroscience.

[50]  Marilyn A. Walker,et al.  An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email , 2000, J. Artif. Intell. Res..

[51]  Carolyn Penstein Rosé,et al.  Tools for Authoring a Dialogue Agent that Participates in Learning Studies , 2007, AIED.

[52]  Kenneth R. Koedinger,et al.  Proceedings of the 2007 conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work , 2007 .

[53]  Roberto Pieraccini,et al.  A stochastic model of computer-human interaction for learning dialogue strategies , 1997, EUROSPEECH.

[54]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[55]  M. Chi,et al.  Eliciting Self‐Explanations Improves Understanding , 1994 .

[56]  Joel R. Tetreault,et al.  Comparing Linguistic Features for Modeling Learning in Computer Tutoring , 2007, AIED.

[57]  Niels Ole Bernsen,et al.  Designing interactive speech systems - from first ideas to user testing , 1998 .

[58]  Kallirroi Georgila,et al.  Hybrid reinforcement/supervised learning for dialogue policies from COMMUNICATOR data , 2005 .

[59]  Kurt VanLehn,et al.  Developing pedagogically effective tutorial dialogue tactics: experiments and a testbed , 2007, SLaTE.

[60]  M. Chi,et al.  Can Tutors Monitor Students' Understanding Accurately? , 2004 .