The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems

This article is a review of experiments comparing the effectiveness of human tutoring, computer tutoring, and no tutoring. “No tutoring” refers to instruction that teaches the same content without tutoring. The computer tutoring systems were divided by their granularity of the user interface interaction into answer-based, step-based, and substep-based tutoring systems. Most intelligent tutoring systems have step-based or substep-based granularities of interaction, whereas most other tutoring systems (often called CAI, CBT, or CAL systems) have answer-based user interfaces. It is widely believed as the granularity of tutoring decreases, the effectiveness increases. In particular, when compared to No tutoring, the effect sizes of answer-based tutoring systems, intelligent tutoring systems, and adult human tutors are believed to be d = 0.3, 1.0, and 2.0 respectively. This review did not confirm these beliefs. Instead, it found that the effect size of human tutoring was much lower: d = 0.79. Moreover, the effect size of intelligent tutoring systems was 0.76, so they are nearly as effective as human tutoring.

[1]  R. Barker Bausell,et al.  A Factorial Study of Tutoring Versus Classroom Instruction1 , 1972 .

[2]  J. Bruner,et al.  The role of tutoring in problem solving. , 1976, Journal of child psychology and psychiatry, and allied disciplines.

[3]  Richard J. Shavelson,et al.  Three Experiments on Learning to Teach , 1976 .

[4]  S G Smith,et al.  Educational uses of the PLATO computer system. , 1976, Science.

[5]  Chen-Lin C. Kulik,et al.  Effectiveness of Computer-based College Teaching: A Meta-analysis of Findings , 1980 .

[6]  Chen-Lin C. Kulik,et al.  Educational Outcomes of Tutoring: A Meta-analysis of Findings , 1982 .

[7]  Frederick Reif,et al.  Prescribing Effective Human Problem-Solving Processes: Problem Description in Physics. Working Paper ES-19. , 1984 .

[8]  J. Kulik,et al.  Effects of Computer-Based Teaching on Secondary School Students. , 1983 .

[9]  B. Bloom.  The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring , 1984 .

[10]  Ann L. Brown,et al.  Reciprocal Teaching of Comprehension-Fostering and Comprehension-Monitoring Activities , 1984 .

[11]  Thomas E. Scruggs,et al.  Tutoring Learning Disabled Students: A Critical Review , 1985 .

[12]  Ralph T. Putnam.  Structuring and Adjusting Content for Students: A Study of Live and Simulated Tutoring of Addition , 1987 .

[13]  Martha C. Polson,et al.  Foundations of intelligent tutoring systems , 1988 .

[14]  Anthony E. Kelly,et al.  Studies of Diagnosis and Remediation with High School Algebra Students , 1989, Cogn. Sci..

[15]  C. Stasz,et al.  Tutoring Techniques in Algebra , 1990 .

[16]  Valerie J. Shute,et al.  A Large-Scale Evaluation of an Intelligent Discovery World: Smithtown , 1990, Interact. Learn. Environ..

[17]  Barbara A. Fox.  Cognitive and interactional aspects of correction in tutoring , 1991 .

[18]  Chen-Lin C. Kulik,et al.  Effectiveness of computer-based instruction: An updated analysis. , 1991 .

[19]  Scott D. Johnson.  Application of Cognitive Theory to the Design, Development, and Implementation of a Computer-Based Troubleshooting Tutor. , 1992 .

[20]  J. Gregory Trafton,et al.  Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems , 1992 .

[21]  Peter Johnson,et al.  Different explanatory dialogue styles and their effects on knowledge acquisition by novices , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[22]  Barbara A. Fox.  The Human Tutorial Dialogue Project: Issues in the Design of instructional Systems , 1993 .

[23]  Scott D. Johnson,et al.  Enhancing Electrical Troubleshooting Skills in a Computer-Coached Practice Environment , 1993, Interact. Learn. Environ..

[24]  John R. Anderson,et al.  Effective Use of Intelligent Software in High School Math Classrooms , 1993 .

[25]  M. Lepper,et al.  Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. , 1993 .

[26]  Harold F. O'Neil,et al.  Meta-analytic studies of findings on computer-based instruction. , 1994 .

[27]  Jim E. Greer,et al.  The VCR Tutor: Effective Instruction for Device Operation , 1995 .

[28]  Alan M. Lesgold,et al.  Tutoring Transfer of Technical Competence. , 1995 .

[29]  D. C. Merrill,et al.  Tutoring: Guided Learning by Doing , 1995 .

[30]  Claire M. Fletcher-Flinn,et al.  The Efficacy of Computer Assisted Instruction (CAI): A Meta-Analysis , 1995 .

[31]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[32]  Joseph P. Magliano,et al.  Collaborative dialogue patterns in naturalistic one-to-one tutoring , 1995 .

[33]  K. A. Ericsson,et al.  Expert and exceptional performance: evidence of maximal adaptation to task constraints. , 1996, Annual review of psychology.

[34]  M. Lepper,et al.  Intrinsic motivation and the process of learning: Beneficial effects of contextualization, personalization, and choice. , 1996 .

[35]  M. Chi.  Constructing Self-Explanations and Scaffolded Explanations in Tutoring , 1996 .

[36]  Tom Routen,et al.  Intelligent Tutoring Systems , 1996, Lecture Notes in Computer Science.

[37]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[38]  Joel A. Michael,et al.  Hinting as a Tactic in One-on-One Tutoring , 1996 .

[39]  Edwin P. Christmann,et al.  Progressive Comparison of the Effects of Computer-Assisted Instruction on the Academic Achievement of Secondary Students. , 1997 .

[40]  Kurt Steuck,et al.  Evaluation of and Authentic Learning Environment for Teaching Scientific Inquiry Skills , 1997 .

[41]  Mary A. Mark,et al.  An Interview Reflection on “Intelligent Tutoring Goes to School in the Big City” , 2015, International Journal of Artificial Intelligence in Education.

[42]  B. Wasik.  Volunteer Tutoring Programs in Reading: A Review , 1998 .

[43]  Patrick Lardieri,et al.  The Advanced Embedded Training System (AETS): An Intelligent Embedded Tutoring System for Tactical Team Training , 1998 .

[44]  J. Michael,et al.  NOVICE VS. EXPERT TUTORS: A COMPARISON OF STYLE , 1999 .

[45]  F. Reif,et al.  Teaching scientific thinking skills: Students and computers coaching each other , 1999 .

[46]  Kurt VanLehn,et al.  Rule-Learning Events in the Acquisition of a Complex Skill: An Evaluation of Cascade , 1999 .

[47]  Joel A. Michael,et al.  An Analysis of Multiple Tutoring Protocols , 2000, Intelligent Tutoring Systems.

[48]  Takashi Yamauchi,et al.  Learning from human tutoring , 2001, Cogn. Sci..

[49]  James C. Lester,et al.  The Case for Social Agency in Computer-Based Teaching: Do Students Learn More Deeply When They Interact With Animated Pedagogical Agents? , 2001 .

[50]  John R. Anderson,et al.  Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes , 2001, CHI.

[51]  Albert T. Corbett,et al.  Cognitive Computer Tutors: Solving the Two-Sigma Problem , 2001, User Modeling.

[52]  Johanna D. Moore,et al.  A comparative evaluation of socratic versus didactic tutoring , 2001 .

[53]  John Edwin Smith,et al.  THE EFFECT OF THE CARNEGIE ALGEBRA TUTOR ON STUDENT ACHIEVEMENT AND ATTITUDE IN INTRODUCTORY HIGH SCHOOL ALGEBRA , 2001 .

[54]  Arthur C. Graesser,et al.  Intelligent Tutoring Systems with Conversational Dialogue , 2001, AI Mag..

[55]  Harold F. O'Neil,et al.  Technology Applications in Education: A Learning View , 2002 .

[56]  Antonija Mitrovic,et al.  KERMIT: A Constraint-Based Tutor for Database Modeling , 2002, Intelligent Tutoring Systems.

[57]  Mark R. Lepper,et al.  The wisdom of practice: Lessons learned from the study of highly effective tutors. , 2002 .

[58]  M. Lepper,et al.  The effects of praise on children's intrinsic motivation: a review and synthesis. , 2002, Psychological bulletin.

[59]  Joel A. Michael,et al.  Classifying Student Initiatives and Tutor Responses in Human Keyboard-to-Keyboard Tutoring Sessions , 2002 .

[60]  Claus Zinn,et al.  The Role of Initiative in Tutorial Dialogue , 2003, EACL.

[61]  Sandra Katz,et al.  Going Beyond the Problem Given: How Human Tutors Use Post-Solution Discussions to Support Transfer , 2003, Int. J. Artif. Intell. Educ..

[62]  K. VanLehn,et al.  Why Do Only Some Events Cause Learning During Human Tutoring? , 2003 .

[63]  Stephanie A. Siler,et al.  Does tutors' use of their knowledge of their students enhance one-to-one tutoring? , 2004 .

[64]  M. Chi,et al.  Can Tutors Monitor Students' Understanding Accurately? , 2004 .

[65]  Scotty D. Craig,et al.  Constructing Knowledge from Dialog in an Intelligent Tutoring System: Interactive Learning, Vicarious Learning, and Pedagogical Agents , 2004 .

[66]  Heather H. Mitchell,et al.  AutoTutor: A tutor with dialogue in natural language , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[67]  Vincent Aleven,et al.  Evaluating the Effectiveness of a Tutorial Dialogue System for Self-Explanation , 2004, Intelligent Tutoring Systems.

[68]  Jung Hee Kim,et al.  Effective Behaviors in a Comparison Between Novice and Expert Algebra Tutors , 2005 .

[69]  Carolyn Penstein Rosé,et al.  A First Evaluation of the Instructional Value of Negotiable Problem Solving Goals on the Exploratory Learning Continuum , 2005, AIED.

[70]  Kurt VanLehn,et al.  The Andes Physics Tutoring System: Lessons Learned , 2005, Int. J. Artif. Intell. Educ..

[71]  H. Chad Lane,et al.  Teaching the tacit knowledge of programming to noviceswith natural language tutoring , 2005, Comput. Sci. Educ..

[72]  Jennifer G. Cromley,et al.  What Do Reading Tutors Do? A Naturalistic Study of More and Less Experienced Tutors in Reading , 2005 .

[73]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[74]  Stellan Ohlsson,et al.  Toward a Computational Model of Expert Tutoring: A First Report , 2006, FLAIRS.

[75]  Scotty D. Craig,et al.  The Deep-Level-Reasoning-Question Effect: The Role of Dialogue and Deep-Level-Reasoning Questions During Vicarious Learning , 2006 .

[76]  Albert T. Corbett,et al.  The Impact on Learning of Generating vs. Selecting Descriptions in Analyzing Algebra Example Solutions , 2006, ICLS.

[77]  Joel A. Michael,et al.  One-on-One Tutoring by Humans and Computers , 2006 .

[78]  Brady Clark,et al.  Responding to Student Uncertainty in Spoken Tutorial Dialogue Systems , 2006, Int. J. Artif. Intell. Educ..

[79]  Antonija Mitrovic,et al.  Facilitating deep learning through self-explanation in an open-ended domain , 2006, Int. J. Knowl. Based Intell. Eng. Syst..

[80]  Carolyn Penstein Rosé,et al.  Spoken Versus Typed Human and Computer Dialogue Tutoring , 2006, Int. J. Artif. Intell. Educ..

[81]  Kurt VanLehn,et al.  A Comparison of Decision-Theoretic, Fixed-Policy and Random Tutorial Action Selection , 2006, Intelligent Tutoring Systems.

[82]  H. Chad Lane,et al.  Teaching Negotiation Skills through Practice and Reflection with Virtual Humans , 2006, Simul..

[83]  Richard Stottler,et al.  A New Generation of Tactical Action Officer Intelligent Tutoring System (ITS) , 2006 .

[84]  Roger Azevedo,et al.  The effect of a human agent’s external regulation upon college students’ hypermedia learning , 2007 .

[85]  Steven Ritter,et al.  What Evidence Matters? A randomized field trial of Cognitive Tutor Algebra I , 2007, ICCE.

[86]  Arthur C. Graesser,et al.  When Are Tutorial Dialogues More Effective Than Reading? , 2007, Cogn. Sci..

[87]  Davide Fossati,et al.  Beyond the code-and-count analysis of tutoring dialogues , 2007, AIED.

[88]  Boya Ma,et al.  Comparative Effectiveness of Carnegie Learning's "Cognitive Tutor Bridge to Algebra" Curriculum: A Report of a Randomized Experiment in the Maui School District. Research Report. , 2007 .

[89]  Arthur C. Graesser,et al.  The effects of speech recognition errors on learner2s contributions, knowledge, emotions, and interaction experience , 2007, SLaTE.

[90]  Sandra Katz,et al.  Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes , 2007, AIED.

[91]  Kurt VanLehn.  Intelligent Tutoring Systems for Continuous, Embedded Assessment , 2007 .

[92]  William R. Penuel,et al.  Effectiveness of Reading and Mathematics Software Products: Findings from the First Student Cohort , 2007 .

[93]  Michael J. Timms.  Using Item Response Theory (IRT) to select hints in an ITS , 2007, AIED.

[94]  Beverley Park Woolf,et al.  Building Intelligent Interactive Tutors , 2008 .

[95]  Kurt VanLehn.  The Interaction Plateau: Answer-Based Tutoring < Step-Based Tutoring = Natural Tutoring , 2008, Intelligent Tutoring Systems.

[96]  Neil T. Heffernan,et al.  Comparing Classroom Problem-Solving with No Feedback to Web-Based Homework Assistance , 2008, Intelligent Tutoring Systems.

[97]  C. Dwyer,et al.  The Future of Assessment : Shaping Teaching and Learning , 2008 .

[98]  Davide Fossati,et al.  The Role of Positive Feedback in Intelligent Tutoring Systems , 2008, ACL.

[99]  Michelene T. H. Chi,et al.  Observing Tutorial Dialogues Collaboratively: Insights About Human Tutoring Effectiveness From Vicarious Learning , 2008, Cogn. Sci..

[100]  Peter M. Hastings,et al.  Research Methods Tutor: Evaluation of a dialogue-based tutoring system in the classroom , 2008, Behavior research methods.

[101]  Neil T. Heffernan,et al.  Expanding the Model-Tracing Architecture: A 3rd Generation Intelligent Tutor for Algebra Symbolization , 2008, Int. J. Artif. Intell. Educ..

[102]  Kristy Elizabeth Boyer,et al.  Balancing Cognitive and Motivational Scaffolding in Tutorial Dialogue , 2008, Intelligent Tutoring Systems.

[103]  Glenn D. Blank,et al.  A pedagogical framework for integrating individual learning style into an intelligent tutoring system , 2008 .

[104]  Daniel C. Moos,et al.  Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? , 2008 .

[105]  Davide Fossati,et al.  Learning Linked Lists: Experiments with the iList System , 2008, Intelligent Tutoring Systems.

[106]  Slavomir Stankov,et al.  TEx-Sys model for building intelligent tutoring systems , 2008, Comput. Educ..

[107]  Sidney K. D'Mello,et al.  Dialogue Modes in Expert Tutoring , 2008, Intelligent Tutoring Systems.

[108]  Diane J. Litman,et al.  Analyzing Dependencies Between Student Certainness States and Tutor Responses in a Spoken Dialogue Corpus , 2008 .

[109]  Vincent Aleven,et al.  Educational Software Features that Encourage and Discourage "Gaming the System" , 2009, AIED.

[110]  Roberto Agodini,et al.  Effectiveness of Reading and Mathematics Software Products: Findings From Two Student Cohorts. NCEE 2009-4041. , 2009 .

[111]  Ryan Shaun Joazeiro de Baker,et al.  Differences Between Intelligent Tutor Lessons, and the Choice to Go Off-Task , 2009, EDM.

[112]  Michelene T. H. Chi,et al.  Active-Constructive-Interactive: A Conceptual Framework for Differentiating Learning Activities , 2009, Top. Cogn. Sci..

[113]  Brent Morgan,et al.  Exploring the deep-level reasoning questions effect during vicarious learning among eighth to eleventh graders in the domains of computer literacy and Newtonian physics , 2009 .

[114]  Neil T. Heffernan,et al.  A Comparison of Traditional Homework to Computer-Supported Homework , 2009 .

[115]  Gary W. Ritter,et al.  The Effectiveness of Volunteer Tutoring Programs for Elementary and Middle School Students: A Meta-Analysis , 2009 .

[116]  Davide Fossati,et al.  I learn from you, you learn from me: How to make iList learn from students , 2009, AIED.

[117]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[118]  Kurt VanLehn,et al.  Learning, Interactional, and Motivational Outcomes in One-to-One Synchronous Computer-mediated versus Face-to-Face Tutoring , 2009, Int. J. Artif. Intell. Educ..

[119]  Fred Phillips,et al.  An Intelligent Tutoring System for the Accounting Cycle: Enhancing Textbook Homework with Artificial Intelligence , 2008 .

[120]  Kasia Muldner,et al.  An analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts , 2011, User Modeling and User-Adapted Interaction.

[121]  A. Renkl,et al.  Can tutors be supported in giving effective explanations , 2010 .

[122]  Beverly Park Woolf,et al.  Student Modeling , 2010, Advances in Intelligent Tutoring Systems.

[123]  David Allbritton,et al.  Squeezing Out Gaming Behavior in a Dialog-Based ITS , 2010, Intelligent Tutoring Systems.

[124]  Kurt VanLehn,et al.  Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies , 2011, User Modeling and User-Adapted Interaction.

[125]  Kurt VanLehn,et al.  An Evaluation of Pedagogical Tutorial Tactics for a Natural Language Tutoring System: A Reinforcement Learning Approach , 2011, Int. J. Artif. Intell. Educ..