A multifactor approach to student model evaluation

Creating student models for Intelligent Tutoring Systems (ITS) in novel domains is often a difficult task. In this study, we outline a multifactor approach to evaluating models that we developed in order to select an appropriate student model for our medical ITS. The combination of areas under the receiver-operator and precision-recall curves, with residual analysis, proved to be a useful and valid method for model selection. We improved on Bayesian Knowledge Tracing with models that treat help differently from mistakes, model all attempts, differentiate skill classes, and model forgetting. We discuss both the methodology we used and the insights we derived regarding student modeling in this novel domain.

[1]  H. J. Arnold.  Introduction to the Practice of Statistics , 1990 .

[2]  Monica C. Jackson,et al.  Introduction to the Practice of Statistics , 2001 .

[3]  Walter F. Stenning,et al.  AN EMPIRICAL STUDY , 2003 .

[4]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[5]  John R. Anderson,et al.  Rules of the Mind , 1993 .

[6]  Glenn Gunzelmann,et al.  Knowledge Tracing and Prediction of Future Trainee Performance , 2006 .

[7]  M. Drenth.  San Juan, Puerto Rico , 2001 .

[8]  Beverly Park Woolf,et al.  Improving Intelligent Tutoring Systems: Using Expectation Maximization to Learn Student Skill Levels , 2006, Intelligent Tutoring Systems.

[9]  Joseph E. Beck,et al.  Using Knowledge Tracing to Measure Student Reading Proficiencies , 2004, Intelligent Tutoring Systems.

[10]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[11]  A. Corbett,et al.  Does Help Help ? A Bayes Net Approach to Modeling Tutor Interventions , 2006 .

[12]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[13]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[14]  John R. Anderson,et al.  Implications of the ACT-R Learning Theory: No Magic Bullets , 2000 .

[15]  Ingrid Zukerman,et al.  Predicting users' requests on the WWW , 1999 .

[16]  William J. J. Roberts,et al.  Revisiting autoregressive hidden Markov modeling of speech signals , 2005, IEEE Signal Processing Letters.

[17]  Rebecca S. Crowley,et al.  An intelligent tutoring system for visual classification problem solving , 2006, Artif. Intell. Medicine.

[18]  KURT VANLEHN.  Bayesian student modeling, user interfaces and feedback : A sensitivity analysis , 2001 .

[19]  Hongchi Shi,et al.  An intelligent distributed environment for active learning , 2001, JERC.

[20]  William J. Clancey.  Methodology for building an intelligent tutoring system , 1987 .

[21]  Charles P. Friedman,et al.  Development of visual diagnostic expertise in pathology -- an information-processing study. , 2003, Journal of the American Medical Informatics Association : JAMIA.

[22]  Rebecca S. Crowley,et al.  A data collection framework for capturing ITS data based on an agent communication standard , 2005 .

[23]  Rebecca S. Crowley,et al.  Research paper: Evaluation of an Intelligent Tutoring System in Pathology: Effects of External Representation on Performance Gains, Metacognition, and Acceptance , 2007, J. Am. Medical Informatics Assoc..

[24]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[26]  Kevin Murphy,et al.  Bayes net toolbox for Matlab , 1999 .

[27]  Jim Reye,et al.  Student Modelling Based on Belief Networks , 2004, Int. J. Artif. Intell. Educ..

[28]  Antonija Mitrovic,et al.  Optimising ITS Behaviour with Bayesian Networks and Decision Theory , 2001 .

[29]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[30]  David Lindley,et al.  Introduction to the Practice of Statistics , 1990, The Mathematical Gazette.

[31]  Christopher M. Schlick,et al.  An Empirical Study of Dynamic Bayesian Networks for User Modeling , 2001 .

[32]  William J. Clancey,et al.  NEOMYCIN: Reconfiguring a Rule-Based Expert System for Application to Teaching , 1981, IJCAI.

[33]  E. Seidemann,et al.  Simultaneously recorded single units in the frontal cortex go through sequences of discrete and stable states in monkeys performing a delayed localization task , 1996, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[34]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[35]  Cristina Conati,et al.  Building and evaluating an intelligent pedagogical agent to improve the effectiveness of an educational game , 2004, IUI '04.

[36]  Ron Kohavi,et al.  The Case against Accuracy Estimation for Comparing Induction Algorithms , 1998, ICML.

[37]  Tom Fawcett,et al.  ROC Graphs: Notes and Practical Considerations for Data Mining Researchers , 2003 .

[38]  Hongchi Shi,et al.  An intelligent distributed environment for active learning , 2001, WWW '01.

[39]  W. Kintsch,et al.  Methods and Tactics in Cognitive Science , 1984 .

[40]  Ryan Shaun Joazeiro de Baker,et al.  Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction , 2005, Graphics Interface.

[41]  Hasmik Mehranian,et al.  Evaluating the Feasibility of Learning Student Models from Data , 2005 .

[42]  Zhendong Niu.  Bayesian student modeling, user interfaces and feedback: A sensitivity analysis , 2001 .