Data calibration for statistical-based assessment in constraint-based tutors

Item Response Theory models for constraint-based intelligent tutoring systems.Data-driven assessment of problem solving tasks.Data filtering criteria for Item Response Theory parameters estimation.Best model fit selection criteria. Intelligent Tutoring Systems (ITSs) are one of a wide range of learning environments, where the main activity is problem solving. One of the most successful approaches for implementing ITSs is Constraint-Based Modeling (CBM). Constraint-based tutors have been successfully used as drill-and-practice environments for learning. More recently CBM tutors have been complemented with a model derived from the field of Psychometrics. The goal of this synergy is to provide CBM tutors with a data-driven and sound mechanism of assessment, which mainly consists in applying the principles of Item Response Theory (IRT). The result of this synergy is, therefore, a formal approach that allows carrying out assessments of performance on problem solving tasks. Several previous studies were conducted proving the validity and utility of this combined approach with small groups of students, in short periods of time and using systems designed specifically for assessment purposes. In this paper, the approach has been extended and adapted to deal with a large set of students who used an ITS over a long period of time. The main research questions addressed in this paper are: (1) Which IRT models are more suitable to be used in a constrained-based tutor? (2) Can data collected from the ITS be used as a source for calibrating the constraints characteristic curves? (3) Which is the best strategy to assemble data for calibration? To answer these questions, we have analyzed three years of data from SQL-Tutor.

[1]  Michalis Nik Xenos.  Prediction and assessment of student behaviour in open and distance education in computers using Bayesian networks , 2004, Comput. Educ..

[2]  Antonija Mitrovic,et al.  Fifteen years of constraint-based tutors: what we have achieved and where we are going , 2011, User Modeling and User-Adapted Interaction.

[3]  Konstantina Chrysafiadi,et al.  Student modeling approaches: A literature review for the last decade , 2013, Expert Syst. Appl..

[4]  Kenneth R. Koedinger,et al.  Teaching the Teacher: Tutoring SimStudent Leads to More Effective Cognitive Tutor Authoring , 2014, International Journal of Artificial Intelligence in Education.

[5]  Russell G. Almond,et al.  An Introduction to Evidence-Centered Design , 2015 .

[6]  S. Ian Robertson,et al.  Problem-solving , 2001, Human Thinking.

[7]  R. Mayer.  Thinking, Problem Solving, Cognition , 1983 .

[8]  David H. Jonassen,et al.  Instructional design models for well-structured and III-structured problem-solving learning outcomes , 1997 .

[9]  Richard E. Mayer,et al.  Computer-Based Assessment of Problem Solving. , 1999 .

[10]  Joel A. Shapiro.  An Algebra Subsystem for Diagnosing Students' Input in a Physics Tutoring System , 2005, Int. J. Artif. Intell. Educ..

[11]  Mingsheng Shang,et al.  A hybrid cognitive assessment based on ontology knowledge map and skills , 2015, Knowl. Based Syst..

[12]  Cristina Conati,et al.  Probabilistic Student Modelling to Improve Exploratory Behaviour , 2004, User Modeling and User-Adapted Interaction.

[13]  Antonija Mitrovic.  Large-scale deployment of three intelligent web-based database tutors , 2006, 28th International Conference on Information Technology Interfaces, 2006..

[14]  G. Boulton‐Lewis.  Teaching for quality learning at university , 2008 .

[15]  Moffat Mathews,et al.  Analysing High-Level Help-Seeking Behaviour in ITSs , 2008, AH.

[16]  R. Almond,et al.  A BRIEF INTRODUCTION TO EVIDENCE-CENTERED DESIGN , 2003 .

[17]  M. Scriven.  The methodology of evaluation , 1966 .

[18]  Stellan Ohlsson,et al.  Constraint-Based Student Modeling , 1994 .

[19]  Somnuk Phon-Amnuaisuk,et al.  Properties of Bayesian student model for INQPRO , 2010, Applied Intelligence.

[20]  Richard E. Mayer,et al.  Problem-solving transfer. , 1996 .

[21]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[22]  XenosMichalis.  Prediction and assessment of student behaviour in open and distance education in computers using Bayesian networks , 2004 .

[23]  Chris Dede,et al.  Interweaving Assessments Into Immersive Authentic Simulations: Design Strategies for Diagnostic and Instructional Insights , 2012 .

[24]  Tanja Mitrovic,et al.  Constraint-based tutors: a success story , 2001, AIED.

[25]  Antonija Mitrovic,et al.  Intelligent Tutors for All: The Constraint-Based Approach , 2007, IEEE Intelligent Systems.

[26]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[27]  De Ayala,et al.  The Theory and Practice of Item Response Theory , 2008 .

[28]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[29]  Takushi Tanaka,et al.  Industrial and Engineering Applications of Artificial Intelligence and Expert Systems , 1995, Lecture Notes in Computer Science.

[30]  Philip Banyard,et al.  Thinking & problem solving , 1991 .

[31]  Antonija Mitrovic,et al.  Optimising ITS Behaviour with Bayesian Networks and Decision Theory , 2001 .

[32]  Antonija Mitrovic,et al.  Experiences in Implementing Constraint-Based Modeling in SQL-Tutor , 1998, Intelligent Tutoring Systems.

[33]  Eduardo Guzmán,et al.  A blended E-learning experience in a course of object oriented programming fundamentals , 2009, Knowl. Based Syst..

[34]  Peter Reimann,et al.  Problem solving learning environments and assessment: A knowledge space theory approach , 2013, Comput. Educ..

[35]  Neil T. Heffernan,et al.  Detecting the Moment of Learning , 2010, Intelligent Tutoring Systems.

[36]  Eduardo Guzmán,et al.  Student Knowledge Diagnosis Using Item Response Theory and Constraint-Based Modeling , 2009, AIED.

[37]  Juan-Diego Zapata-Rivera.  Indirectly Visible Bayesian Student Models , 2007, BMA.

[38]  P. Winne,et al.  Handbook of educational psychology , 2015 .

[39]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[40]  Antonija Mitrovic,et al.  A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling , 2003, User Modeling.

[41]  Antonija Mitrovic,et al.  Revisiting Ill-Definedness and the Consequences for ITSs , 2009, AIED.

[42]  P. Johnson-Laird,et al.  The Blackwell dictionary of cognitive psychology , 1990 .

[43]  M. Taras,et al.  ASSESSMENT – SUMMATIVE AND FORMATIVE – SOME THEORETICAL REFLECTIONS , 2005 .

[44]  Antonija Mitrovic,et al.  Evaluation of a Constraint-Based Tutor for a Database Language , 1999 .

[45]  José-Luis Pérez-de-la-Cruz,et al.  Bayesian networks for student model engineering , 2010, Comput. Educ..

[46]  Eduardo Guzmán,et al.  Data-Driven Student Knowledge Assessment through Ill-Defined Procedural Tasks , 2009, CAEPIA.

[47]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[48]  Russell G. Almond,et al.  Bayesian Networks in Educational Assessment , 2015 .

[49]  José-Luis Pérez-de-la-Cruz,et al.  A Bayesian Diagnostic Algorithm for Student Modeling and its Evaluation , 2002, User Modeling and User-Adapted Interaction.

[50]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[51]  Russell G. Almond,et al.  A Brief Introduction to Evidence-Centered Design. CSE Report 632. , 2004 .

[52]  Gulsen Tasdelen Teker,et al.  Comparison of classical test theory and item response theory in terms of item parameters 1 , 2014 .

[53]  Antonija Mitrovic,et al.  Implementing CBM: SQL-Tutor After Fifteen Years , 2015, International Journal of Artificial Intelligence in Education.

[54]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[55]  R. Hambleton,et al.  An NCME Instructional Module on Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development. , 2005 .

[56]  Robert J. Mislevy,et al.  Design Patterns for Model-Based Reasoning , 2017 .

[57]  John Self,et al.  The defining characteristics of intelligent tutoring systems research: ITSs care, precisely , 1998 .

[58]  Stellan Ohlsson,et al.  Learning from Performance Errors. , 1996 .

[59]  Robert J. Mislevy,et al.  Implications of Evidence‐Centered Design for Educational Testing , 2007 .

[60]  M. Scheerer,et al.  Problem Solving , 1967, Nature.

[61]  Peter F. Halpin,et al.  COLLABORATIVE PROBLEM SOLVING AND THE ASSESSMENT OF COGNITIVE SKILLS: PSYCHOMETRIC CONSIDERATIONS , 2013 .

[62]  Robert J. Mislevy,et al.  An evidence centered design for learning and assessment in the digital world , 2010 .