Example-Tracing Tutors: Intelligent Tutor Development for Non-programmers

In 2009, we reported on a new Intelligent Tutoring Systems (ITS) technology, example-tracing tutors, that can be built without programming using the Cognitive Tutor Authoring Tools (CTAT). Creating example-tracing tutors was shown to be 4–8 times as cost-effective as estimates for ITS development from the literature. Since 2009, CTAT and its associated learning management system, the Tutorshop, have been extended and have been used for both research and real-world instruction. As evidence that example-tracing tutors are an effective and mature ITS paradigm, CTAT-built tutors have been used by approximately 44,000 students and account for 40 % of the data sets in DataShop, a large open repository for educational technology data sets. We review 18 example-tracing tutors built since 2009, which have been shown to be effective in helping students learn in real educational settings, often with large pre/post effect sizes. These tutors support a variety of pedagogical approaches, beyond step-based problem solving, including collaborative learning, educational games, and guided invention activities. CTAT and other ITS authoring tools illustrate that non-programmer approaches to building ITS are viable and useful and will likely play a key role in making ITS widespread.

[1]  Vincent Aleven A is for Adaptivity, But What is Adaptivity? Re-Defining the Field of AIED , 2015, AIED Workshops.

[2]  Lisa M. Rossi,et al.  Design Recommendations for Adaptive Intelligent Tutoring Systems Learner Modeling ( Volume I ) 155 CHAPTER 14 ‒ Assessing the Disengaged Behaviors of Learners , 2013 .

[3]  Vincent Aleven,et al.  Rule-Based Cognitive Modeling for Intelligent Tutoring Systems , 2010, Advances in Intelligent Tutoring Systems.

[4]  Judy Kay,et al.  Proceedings of the 15th international conference on Artificial intelligence in education , 2011 .

[5]  Bruce M. McLaren,et al.  Evaluating a Bayesian Student Model of Decimal Misconceptions , 2011, EDM.

[6]  Antonija Mitrovic,et al.  ASPIRE: An Authoring System and Deployment Environment for Constraint-Based Tutors , 2009, Int. J. Artif. Intell. Educ..

[7]  David F. Feldon,et al.  Cognitive task analysis , 2009 .

[8]  Stephen B. Gilbert,et al.  Expansion of the xPST Framework to Enable Non-programmers to Create Intelligent Tutoring Systems in 3D Game Environments , 2010, Intelligent Tutoring Systems.

[9]  David H. Jonassen,et al.  Handbook of Research for educational Communications and Technology , 1997 .

[10]  Vincent Aleven,et al.  Supporting Students' Self-Regulated Learning with an Open Learner Model in a Linear Equation Tutor , 2013, AIED.

[11]  Albert T. Corbett,et al.  Modeling Student Knowledge: Cognitive Tutors in High School and College , 2000, User Modeling and User-Adapted Interaction.

[12]  Cornelia S. Große,et al.  How Fading Worked Solution Steps Works – A Cognitive Load Perspective , 2004 .

[13]  Stephen B. Gilbert,et al.  Evaluation of Two Intelligent Tutoring System Authoring Tool Paradigms: Graphical User Interface-Based and Text-Based , 2012 .

[14]  Carolyn Penstein Rosé,et al.  The Beginning of a Beautiful Friendship? Intelligent Tutoring Systems and MOOCs , 2015, AIED.

[15]  Vincent Aleven,et al.  The Design of an Exploratory Learning Environment to Support Invention , 2015, AIED Workshops.

[16]  Antonija Mitrovic,et al.  Evaluation of a Constraint-Based Tutor for a Database Language , 1999 .

[17]  Richard E. Mayer,et al.  Polite web-based intelligent tutors: Can they improve learning in classrooms? , 2011, Comput. Educ..

[18]  Albert T. Corbett,et al.  The Knowledge-Learning-Instruction Framework: Bridging the Science-Practice Chasm to Enhance Robust Student Learning , 2012, Cogn. Sci..

[19]  Vincent Aleven,et al.  Does supporting multiple student strategies lead to greater learning and motivation? Investigating a source of complexity in the architecture of intelligent tutoring systems , 2013, Comput. Educ..

[20]  Beverly Park Woolf,et al.  Building Intelligent Interactive Tutors: Student-centered Strategies for Revolutionizing E-learning , 2008 .

[21]  Tom Murray,et al.  Authoring Intelligent Tutoring Systems: An analysis of the state of the art , 1999 .

[22]  Vincent Aleven,et al.  Gamification of Joint Student/System Control over Problem Selection in a Linear Equation Tutor , 2014, Intelligent Tutoring Systems.

[23]  R. Atkinson,et al.  Transitioning From Studying Examples to Solving Problems: Effects of Self-Explanation Prompts and Fading Worked-Out Steps. , 2003 .

[24]  Vincent Aleven,et al.  Instruction Based on Adaptive Learning Technologies , 2016 .

[25]  Slava Kalyuga Expertise Reversal Effect and Its Implications for Learner-Tailored Instruction , 2007 .

[26]  Vincent Aleven,et al.  Towards the Development of the Invention Coach: a Naturalistic Study of Teacher Guidance for an Exploratory Learning Task , 2015, AIED.

[27]  Vincent Aleven,et al.  Replay Analysis in Open-Ended Educational Games , 2015 .

[28]  Joseph E. Beck,et al.  Macroadapting Animalwatch to Gender and Cognitive Differnces with Respect to Hint Interactivity and Symbolism , 2000, Intelligent Tutoring Systems.

[29]  Jean-François Lebeau,et al.  Authoring Problem-Solving Tutors: A Comparison between ASTUS and CTAT , 2010, Advances in Intelligent Tutoring Systems.

[30]  Kenneth R. Koedinger,et al.  A Cognitive Task Analysis of Using Pictures To Support Pre-Algebraic Reasoning , 2002 .

[31]  Rohit Kumar,et al.  Special Issue on Intelligent Support for Learning in Groups , 2013, International Journal of Artificial Intelligence in Education.

[32]  Vincent Aleven,et al.  The Invention Lab: Using a Hybrid of Model Tracing and Constraint-Based Modeling to Offer Intelligent Support in Inquiry Environments , 2010, Intelligent Tutoring Systems.

[33]  Arthur C. Graesser,et al.  AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring , 2014, International Journal of Artificial Intelligence in Education.

[34]  Martina A. Rau,et al.  Connection making between multiple graphical representations: A multi-methods approach for domain-specific grounding of an intelligent tutoring system for chemistry , 2015, Comput. Educ..

[35]  Daniel F. McCaffrey,et al.  Effectiveness of Cognitive Tutor Algebra I at Scale , 2014 .

[36]  Stephen B. Gilbert,et al.  Evaluation of WebxPST: A Browser-Based Authoring Tool for Problem-Specific Tutors , 2011, AIED.

[37]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[38]  Jean McKendree,et al.  Effective Feedback Content for Tutoring Complex Skills , 1990, Hum. Comput. Interact..

[39]  Kurt VanLehn Regulative Loops, Step Loops and Task Loops , 2015, International Journal of Artificial Intelligence in Education.

[40]  Ido Roll,et al.  Making the failure more productive: scaffolding the invention process to improve inquiry behaviors and outcomes in invention activities , 2014 .

[41]  Antonija Mitrovic,et al.  Supporting collaborative learning and problem-solving in a constraint-based CSCL environment for UML class diagrams , 2007, Int. J. Comput. Support. Collab. Learn..

[42]  R. Slavin Research on cooperative learning and achievement: What we know, what we need to know. , 1996 .

[43]  A. Corbett,et al.  The Cambridge Handbook of the Learning Sciences: Cognitive Tutors , 2005 .

[44]  Vincent Aleven,et al.  A New Paradigm for Intelligent Tutoring Systems: Example-Tracing Tutors , 2009, Int. J. Artif. Intell. Educ..

[45]  Judy Kay,et al.  MOOCs: So Many Learners, So Much Potential ... , 2013, IEEE Intelligent Systems.

[46]  Mitchell J. Nathan Knowledge and Situational Feedback in a Learning Environment for Algebra Story Problem Solving , 1998, Interact. Learn. Environ..

[47]  Slava Kalyuga,et al.  The Expertise Reversal Effect , 2003 .

[48]  Stephen B. Gilbert,et al.  From SDK to xPST: A New Way to Overlay a Tutor on Existing Software , 2009, FLAIRS Conference.

[49]  Kenneth R. Koedinger,et al.  Adaptive Intelligent Support to Improve Peer Tutoring in Algebra , 2013, International Journal of Artificial Intelligence in Education.

[50]  K. Koedinger,et al.  Exploring the Assistance Dilemma in Experiments with Cognitive Tutors , 2007 .

[51]  Kenneth R. Koedinger,et al.  A Data Repository for the EDM Community: The PSLC DataShop , 2010 .

[52]  Connie Qun Guan,et al.  Writing strengthens orthography and alphabetic-coding strengthens phonology in learning to read Chinese , 2011 .

[53]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[54]  Kevin D. Ashley,et al.  A case-based approach to modeling legal expertise , 1988, IEEE Expert.

[55]  Ryan Shaun Joazeiro de Baker,et al.  The Difficulty Factors Approach to the Design of Lessons in Intelligent Tutor Curricula , 2007, Int. J. Artif. Intell. Educ..

[56]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[57]  Kenneth R. Koedinger,et al.  Conceptual Scaffolding to Check One's Procedures , 2013, AIED.

[58]  Arthur C. Graesser,et al.  Introduction to the special issue on advanced learning technologies , 2013 .

[59]  Daniel L. Schwartz,et al.  Inventing to Prepare for Future Learning: The Hidden Efficiency of Encouraging Original Student Production in Statistics Instruction , 2004 .

[60]  Carolyn Penstein Rosé,et al.  Towards an Agile Approach to Adapting Dynamic Collaboration Support to Student Needs , 2014, International Journal of Artificial Intelligence in Education.

[61]  Tamara van Gog,et al.  Exploring the Assistance Dilemma: Comparing Instructional Support in Examples and Problems , 2014, Intelligent Tutoring Systems.

[62]  K. Koedinger,et al.  Accounting for Beneficial Effects of Worked Examples in Tutored Problem Solving , 2010 .

[63]  Judy Kay,et al.  Open Learner Models , 2010, Advances in Intelligent Tutoring Systems.

[64]  Alexander Renkl,et al.  Toward an Instructionally Oriented Theory of Example-Based Learning , 2014, Cogn. Sci..

[65]  Kenneth R. Koedinger,et al.  When and how often should worked examples be given to students? New results and a summary of the current state of research , 2008 .

[66]  Kenneth R. Koedinger,et al.  Analogies, Explanations, and Practice: Examining How Task Types Affect Second Language Grammar Learning , 2010, Intelligent Tutoring Systems.

[67]  David Rosenthal,et al.  An Assessment of Constraint-Based Tutors: A Response to Mitrovic and Ohlsson's Critique of "A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms" , 2006, Int. J. Artif. Intell. Educ..

[68]  John Sweller,et al.  Cognitive Load Theory: Instructional Implications of the Interaction between Information Structures and Cognitive Architecture , 2004 .

[69]  Yvonne Earnshaw,et al.  Effects of Levels of Instructional Assistance on Learning and Mental Effort in an Intelligent Tutoring System , 2014 .

[70]  Jean-François Lebeau,et al.  Designing a Knowledge Representation Approach for the Generation of Pedagogical Interventions by MTTs , 2014, International Journal of Artificial Intelligence in Education.

[71]  Kenneth R. Koedinger,et al.  When seeing isn't believing: Influences of prior conceptions and misconceptions , 2013, CogSci.

[72]  B. MacWhinney,et al.  Learning a Tonal Language by Attending to the Tone: An In Vivo Experiment , 2011 .

[73]  T. Gog,et al.  Example-Based Learning: Integrating Cognitive and Social-Cognitive Research Perspectives , 2010 .

[74]  Cristina Conati,et al.  Toward Computer-Based Support of Meta-Cognitive Skills: a Computational Framework to Coach Self-Explanation , 2000 .

[75]  Steven Ritter,et al.  Authoring Model-Tracing Cognitive Tutors , 2009, Int. J. Artif. Intell. Educ..

[76]  Vincent Aleven,et al.  Toward Meta-cognitive Tutoring: A Model of Help Seeking with a Cognitive Tutor , 2006, Int. J. Artif. Intell. Educ..

[77]  Martina A. Rau,et al.  Why Do the Rich Get Richer? A Structural Equation Model to Test How Spatial Skills Affect Learning with Representations , 2015, EDM.

[78]  B. McLaren,et al.  Decimal Point : Designing and Developing an Educational Game to Teach Decimals to Middle School Students , 2014 .

[79]  Vincent Aleven,et al.  Eliciting Intelligent Novice Behaviors with Grounded Feedback in a Fraction Addition Tutor , 2011, AIED.

[80]  K. Koedinger,et al.  Designing Knowledge Scaffolds to Support Mathematical Problem Solving , 2005 .

[81]  John R. Anderson,et al.  Skill Acquisition and the LISP Tutor , 1989, Cogn. Sci..

[82]  Vincent Aleven,et al.  Investigating Effects of Embedding Collaboration in an Intelligent Tutoring System for Elementary School Students. , 2016 .

[83]  Peter Maurer,et al.  The Cambridge Handbook of the Learning Sciences , 2022 .

[84]  Zachary A. Pardos,et al.  How Should Intelligent Tutoring Systems Sequence Multiple Graphical Representations of Fractions? A Multi-Methods Study , 2013, International Journal of Artificial Intelligence in Education.

[85]  Kenneth R. Koedinger,et al.  An architecture for plug-in tutor agents , 1996 .

[86]  F. Paas,et al.  Variability of Worked Examples and Transfer of Geometrical Problem-Solving Skills: A Cognitive-Load Approach , 1994 .

[87]  Franceska Xhakaj,et al.  Teaching a Complex Process: Insertion in Red Black Trees , 2015, AIED.

[88]  Vincent Aleven,et al.  Using an Intelligent Tutoring System to Support Collaborative as well as Individual Learning , 2014, Intelligent Tutoring Systems.

[89]  Franceska Xhakaj,et al.  A New Approach To Teaching Red Black Tree , 2015, ITiCSE.

[90]  Vincent Aleven,et al.  Successful learning with multiple graphical representations and self-explanation prompts. , 2015 .

[91]  Fabienne M. Van der Kleij,et al.  Effects of Feedback in a Computer-Based Learning Environment on Students’ Learning Outcomes , 2013 .

[92]  Rice,et al.  Moodle : E-learning course development : a complete guide to successful learning using Moodle , 2006 .

[93]  Herbert A. Simon,et al.  Computer Science as Empirical Inquiry , 2011 .

[94]  Albert T. Corbett,et al.  A Cognitive Tutor for Genetics Problem Solving: Learning Gains and Student Modeling , 2010 .

[95]  Vincent Aleven,et al.  The Cognitive Tutor Authoring Tools (CTAT): Preliminary Evaluation of Efficiency Gains , 2006, Intelligent Tutoring Systems.

[96]  Tom Murray,et al.  Authoring tools for advanced technology learning environments : toward cost-effective adaptive, interactive and intelligent educational software , 2003 .

[97]  Richard E. Mayer,et al.  To Err Is Human, to Explain and Correct Is Divine: A Study of Interactive Erroneous Examples with Middle School Math Students , 2012, EC-TEL.

[98]  Kenneth R. Koedinger,et al.  Teaching the Teacher: Tutoring SimStudent Leads to More Effective Cognitive Tutor Authoring , 2014, International Journal of Artificial Intelligence in Education.

[99]  Vincent Aleven,et al.  Towards Understanding How to Leverage Sense-making, Induction/Refinement and Fluency to Improve Robust Learning , 2015, EDM.

[100]  Marsha C. Lovett,et al.  Cognotive Task Analysis in Service of Intelligent Tutoring System Design: A Case Study in Statistics , 1998, Intelligent Tutoring Systems.

[101]  Tom Murray,et al.  Authoring Tools for Advanced Technology Learning Environments , 2003 .

[102]  Neil T. Heffernan,et al.  The ASSISTment Builder: Supporting the Life Cycle of Tutoring System Content Creation , 2009, IEEE Transactions on Learning Technologies.

[103]  K. Koedinger,et al.  Fostering the Intelligent Novice: Learning From Errors With Metacognitive Tutoring , 2005 .

[104]  Kalyan Veeramachaneni,et al.  Transfer Learning for Predictive Models in Massive Open Online Courses , 2015, AIED.

[105]  Kenneth R. Koedinger,et al.  Effects of Adaptive Prompted Self-explanation on Robust Learning of Second Language Grammar , 2011, AIED.

[106]  Zachary A. Pardos,et al.  Understanding Student Success in Chemistry Using Gaze Tracking and Pupillometry , 2015, AIED.

[107]  J. Janssen,et al.  Coordinated Computer-Supported Collaborative Learning: Awareness and Awareness Tools , 2013 .

[108]  Daniel L. Schwartz,et al.  Practicing versus inventing with contrasting cases: The effects of telling first on learning and transfer. , 2011 .

[109]  Richard E. Mayer,et al.  A politeness effect in learning with web-based intelligent tutors , 2011, Int. J. Hum. Comput. Stud..

[110]  V. Aleven,et al.  Rapid Authoring of Intelligent Tutors for Real-World and Experimental Use , 2006, Sixth IEEE International Conference on Advanced Learning Technologies (ICALT'06).

[111]  Neil T. Heffernan,et al.  The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and Teaching , 2014, International Journal of Artificial Intelligence in Education.

[112]  Vincent Aleven,et al.  Advancing Development of Intercultural Competence through Supporting Predictions in Narrative Video , 2009, Int. J. Artif. Intell. Educ..

[113]  Allen Newell,et al.  Computer science as empirical inquiry: symbols and search , 1976, CACM.

[114]  Julie L. Booth,et al.  Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples , 2013 .

[115]  Richard E. Mayer,et al.  Delayed Learning Effects with Erroneous Examples: a Study of Learning Decimals with a Web-Based Tutor , 2015, International Journal of Artificial Intelligence in Education.

[116]  Tamara van Gog,et al.  Worked Examples are More Efficient for Learning than High-Assistance Instructional Software , 2015, AIED.

[117]  Daniel D. Suthers,et al.  Component-Based Construction of a Science Learning Space , 1998, Intelligent Tutoring Systems.

[118]  Richard E. Mayer,et al.  Using erroneous examples to improve mathematics learning with a web-based tutoring system , 2014, Comput. Hum. Behav..

[119]  Vincent Aleven,et al.  Intelligent Tutoring Goes To School in the Big City , 1997 .

[120]  Kenneth R. Koedinger Toward Evidence for Instructional Design Principles: Examples from Cognitive Tutor Math 6 , 2002 .

[121]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[122]  Kenneth R. Koedinger,et al.  Authoring Tutors with SimStudent: An Evaluation of Efficiency and Model Quality , 2014, Intelligent Tutoring Systems.

[123]  Vincent Aleven,et al.  Authoring Tools for Collaborative Intelligent Tutoring System Environments , 2014, Intelligent Tutoring Systems.

[124]  Vincent Aleven,et al.  Active Learners: Redesigning an Intelligent Tutoring System to Support Self-regulated Learning , 2013, EC-TEL.

[125]  Tamara van Gog,et al.  The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in computer-based learning environments , 2016, Comput. Hum. Behav..

[126]  Bruce M. McLaren,et al.  How Much Assistance Is Helpful to Students in Discovery Learning? , 2009, EC-TEL.

[127]  Kenneth R. Koedinger,et al.  Is Self-Explanation Always Better? The Effects of Adding Self-Explanation Prompts to an English Grammar Tutor , 2009 .

[128]  Vincent Aleven,et al.  Metacognitive Practice Makes Perfect: Improving Students' Self-Assessment Skills with an Intelligent Tutoring System , 2011, AIED.

[129]  Vincent Aleven,et al.  The expertise reversal effect and worked examples in tutored problem solving , 2010 .

[130]  Tobias Ley,et al.  Scaling up Learning for Sustained Impact , 2013, Lecture Notes in Computer Science.

[131]  K. VanLehn The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems , 2011 .

[132]  Albert T. Corbett,et al.  Cognitive Tutor: Applied research in mathematics education , 2007, Psychonomic bulletin & review.

[133]  Martina A. Rau,et al.  ITS Support for Conceptual and Perceptual Connection Making Between Multiple Graphical Representations , 2015, AIED.

[134]  Robert A. Sottilare,et al.  Considerations in the Development of an Ontology for a Generalized Intelligent Framework for Tutoring , 2012 .

[135]  Vincent Aleven,et al.  Scaling Up Programming by Demonstration for Intelligent Tutoring Systems Development: An Open-Access Web Site for Middle School Mathematics Learning , 2009, IEEE Transactions on Learning Technologies.

[136]  Vincent Aleven,et al.  Interleaved Practice in Multi-Dimensional Learning Tasks: Which Dimension Should We Interleave?. , 2013 .

[137]  Slavomir Stankov,et al.  TEx-Sys model for building intelligent tutoring systems , 2008, Comput. Educ..

[138]  Neil T. Heffernan,et al.  Opening the Door to Non-programmers: Authoring Intelligent Tutor Behavior by Demonstration , 2004, Intelligent Tutoring Systems.

[139]  Judy Kay,et al.  MOOClm: User Modelling for MOOCs , 2015, UMAP.

[140]  Vincent Aleven,et al.  Different Futures of Adaptive Collaborative Learning Support , 2016, International Journal of Artificial Intelligence in Education.

[141]  Kenneth R. Koedinger,et al.  Extending the self-explanation effect to second language grammar learning , 2010, ICLS.

[142]  Vincent Aleven,et al.  Sense Making Alone Doesn't Do It: Fluency Matters Too! ITS Support for Robust Learning with Multiple Representations , 2012, ITS.

[143]  Stephen B. Gilbert,et al.  Evaluating an Authoring Tool for Model-Tracing Intelligent Tutoring Systems , 2008, Intelligent Tutoring Systems.

[144]  Kasia Muldner,et al.  A Multimedia Adaptive Tutoring System for Mathematics that Addresses Cognition, Metacognition and Affect , 2014, International Journal of Artificial Intelligence in Education.