Statistical Techniques to Explore the Quality of Constraints in Constraint-Based Modeling Environments

One of the most popular student modeling approaches is Constraint-Based Modeling (CBM). It is an efficient approach that can be easily applied inside an Intelligent Tutoring System (ITS). Even with these characteristics, building new ITSs requires carefully designing the domain model to be taught because different sources of errors could affect the efficiency of the system. In this paper a novel mechanism for studying the quality of the elements in the domain model of CBM systems is presented. This mechanism combines CBM with the Item Response Theory (IRT), a data-driven technique for automatic assessment. The goal is to improve the quality of the elements that are used in problem solving environments for assessment or instruction. In this paper we propose a set of statistical techniques, i.e., the analysis of the point-biserial correlation, the Cronbach’s alpha and the information function, to explore the quality of constraints. Two different tools have been used to test this approach: a problem solving environment designed to assess students in project investment analysis; and an independent component that performs assessments using CBM and IRT. Results suggest that the three methods produce consistent diagnosis and may be complementary in some cases. In the experiments we have carried out they were able to detect faulty, bad and good quality constraints.

[1]  Kenneth R. Koedinger,et al.  Performance Factors Analysis - A New Alternative to Knowledge Tracing , 2009, AIED.

[2]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[3]  Christine E. DeMars,et al.  Item Response Theory , 2010, Assessing Measurement Invariance for Applied Research.

[4]  Ricardo Conejo,et al.  SIETTE: A Web-Based Tool for Adaptive Testing , 2004, Int. J. Artif. Intell. Educ..

[5]  R. Almond,et al.  A BRIEF INTRODUCTION TO EVIDENCE-CENTERED DESIGN , 2003 .

[6]  L. L. Thurstone,et al.  A method of scaling psychological and educational tests. , 1925 .

[7]  Antonija Mitrovic,et al.  Optimising ITS Behaviour with Bayesian Networks and Decision Theory , 2001 .

[8]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[9]  Antonija Mitrovic,et al.  Using Evaluation to Shape ITS Design: Results and Experiences with SQL-Tutor , 2002, User Modeling and User-Adapted Interaction.

[10]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[11]  Eduardo Guzmán,et al.  Student Knowledge Diagnosis Using Item Response Theory and Constraint-Based Modeling , 2009, AIED.

[12]  Antonija Mitrovic,et al.  Constraint-based knowledge representation for individualized instruction , 2006, Comput. Sci. Inf. Syst..

[13]  Stellan Ohlsson,et al.  Learning from Performance Errors. , 1996 .

[14]  Antonija Mitrovic,et al.  Intelligent Tutors for All: The Constraint-Based Approach , 2007, IEEE Intelligent Systems.

[15]  Eduardo Guzmán,et al.  A SOA-Based Framework for Constructing Problem Solving Environments , 2008, 2008 Eighth IEEE International Conference on Advanced Learning Technologies.

[16]  Joseph Lev,et al.  The Point Biserial Coefficient of Correlation , 1949 .

[17]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[18]  Antonija Mitrovic,et al.  Using Learning Curves to Mine Student Models , 2005, User Modeling.

[19]  Stellan Ohlsson,et al.  Constraint-Based Student Modeling , 1994 .

[20]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[21]  Eduardo Guzmán,et al.  Exploring Quality of Constraints for Assessment in Problem Solving Environments , 2012, ITS.

[22]  Jaime Galvez Cordero,et al.  Exploring quality of constraints for assessment in problem solving environments , 2012, ITS 2012.

[23]  Zachary A. Pardos,et al.  KT-IDEM: introducing item difficulty to the knowledge tracing model , 2011, UMAP'11.

[24]  Christine E. DeMars.  Item Response Theory , 2010 .

[25]  Antonija Mitrovic,et al.  The Effect of Adapting Feedback Generality in ITS , 2006, AH.

[26]  Antonija Mitrovic,et al.  Evaluating and improving adaptive educational systems with learning curves , 2011, User Modeling and User-Adapted Interaction.

[27]  Gilah C. Leder.  Assessment and learning of mathematics , 1992 .

[28]  Melvin R. Novick,et al.  Statistical Theories of Mental Test Scores , 1969 .

[29]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[30]  Kenneth R. Koedinger,et al.  Comparing Two IRT Models for Conjunctive Skills , 2008, Intelligent Tutoring Systems.

[31]  F. Baker.  The basics of item response theory , 1985 .

[32]  Eduardo Guzmán,et al.  Improving Student Performance Using Self-Assessment Tests , 2007, IEEE Intelligent Systems.

[33]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[34]  Antonija Mitrovic,et al.  Authoring Constraint-Based Tutors in ASPIRE , 2006, Intelligent Tutoring Systems.

[35]  J. P. Meyer,et al.  Fair and Equitable Measurement of Student Learning in MOOCS: An Introduction to Item Response Theory, Scale Linking, and Score Equating. , 2013 .

[36]  Eduardo Guzmán,et al.  Data-Driven Student Knowledge Assessment through Ill-Defined Procedural Tasks , 2009, CAEPIA.

[37]  Antonija Mitrovic,et al.  A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling , 2003, User Modeling.

[38]  Antonija Mitrovic,et al.  Fifteen years of constraint-based tutors: what we have achieved and where we are going , 2011, User Modeling and User-Adapted Interaction.

[39]  Antonija Mitrovic,et al.  Revisiting Ill-Definedness and the Consequences for ITSs , 2009, AIED.