Diagnosis of Subtraction Bugs Using Bayesian Networks

Diagnosis of misconceptions or ‘‘bugs’’ in procedural skills is difficult because of their unstable nature. This study addresses this problem by proposing and evaluating a probability-based approach to the diagnosis of bugs in children’s multicolumn subtraction performance using Bayesian networks. This approach assumes a causal network relating hypothesized subtraction bugs to the observed test items. Two research questions are tested within this framework. First, it is investigated whether more reliable assessment of latent subtraction bugs can be achieved by hypothesizing and using subskill nodes in the Bayesian network as causal factors affecting bugs. Second, network performance is evaluated using two types of testing situations, one using binary data (items scored as correct or incorrect) and the other simulating a multiple-choice test format with diagnostic use of specific wrong answers. The resulting four types of Bayesian networks are evaluated for their effectiveness in bug diagnosis. All four networks show good performance, with even the simplest network (bug nodes only, binary data) giving overall bug diagnosis rates of at least 85%. Prediction is best with the most complex network (bug and subskill nodes, diagnostic use of specific wrong answers), for which the correct diagnosis rate reaches 99%. These results suggest that stable and reliable bug diagnosis can be achieved using a Bayesian network framework, but that the stability and effectiveness of diagnosis is increased when the network includes latent subskills in addition to bugs as causal factors, and when specific wrong answers are used for diagnostic purposes.

[1]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[2]  Marek J. Druzdzel,et al.  Qualtitative propagation and scenario-based scheme for exploiting probabilistic reasoning , 1990, UAI.

[3]  David Heckerman,et al.  A combination of cutset conditioning with clique-tree propagation in the Pathfinder system , 1990, UAI.

[4]  Bruce D'Ambrosio,et al.  Local expression languages for probabilistic dependence , 1995, Int. J. Approx. Reason..

[5]  Michelene T. H. Chi,et al.  Commonsense Conceptions of Emergent Processes: Why Some Misconceptions Are Robust , 2005 .

[6]  John Seely Brown,et al.  Diagnostic Models for Procedural Bugs in Basic Mathematical Skills , 1978, Cogn. Sci..

[7]  Robert J. Mislevy,et al.  PROBABILITY-BASED INFERENCE IN A DOMAIN OF PROPORTIONAL REASONING TASKS , 1992 .

[8]  K. Sijtsma,et al.  Knowledge of Solution Strategies and IRT Modeling of Items for Transitive Reasoning , 1999 .

[9]  David E. Brown,et al.  Overcoming misconceptions via analogical reasoning: abstract transfer versus explanatory model construction , 1989 .

[10]  Maja O’Connor.  LONGITUDINAL STUDY , 2013 .

[11]  Susan E. Embretson,et al.  A multidimensional latent trait model for measuring learning and change , 1991 .

[12]  Klaas Sijtsma,et al.  Methodology Review: Nonparametric IRT Approaches to the Analysis of Dichotomous Item Scores , 1998 .

[13]  Kikumi K. Tatsuoka,et al.  A Probabilistic Model for Diagnosing Misconceptions By The Pattern Classification Approach , 1985 .

[14]  Jeffrey A Douglas,et al.  Higher-order latent trait models for cognitive diagnosis , 2004 .

[15]  Lakhmi C. Jain,et al.  Introduction to Bayesian Networks , 2008 .

[16]  Kurt VanLehn,et al.  Repair Theory: A Generative Theory of Bugs in Procedural Skills , 1980, Cogn. Sci..

[17]  Susan B. Empson,et al.  A longitudinal study of invention and understanding in children's multidigit addition and subtraction , 1998 .

[18]  Zhendong Niu.  Bayesian student modeling, user interfaces and feedback: A sensitivity analysis , 2001 .

[19]  Matthias von Davier,et al.  PARTIALLY OBSERVED MIXTURES OF IRT MODELS: AN EXTENSION OF THE GENERALIZED PARTIAL CREDIT MODEL , 2003 .

[20]  Richard M. Young,et al.  Errors in Children's Subtraction , 1981, Cogn. Sci..

[21]  Robert J. Mislevy,et al.  Diagnostic Assessment of Troubleshooting Skill in an Intelligent Tutoring System. , 1994 .

[22]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[23]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[24]  Kristian G. Olesen,et al.  HUGIN - A Shell for Building Bayesian Belief Universes for Expert Systems , 1989, IJCAI.

[25]  R. Almond,et al.  Making Sense of Data from Complex Assessments. CSE Technical Report. , 2001 .

[26]  M. Chi,et al.  Can Tutors Monitor Students' Understanding Accurately? , 2004 .

[27]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[28]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1998 .

[29]  Michel C. Desmarais,et al.  Computer Adaptive Testing: Comparison of a Probabilistic Network Approach with Item Response Theory , 2005, User Modeling.

[30]  Derek H. Sleeman,et al.  An Attempt to Understand Students' Understanding of Basic Algebra , 1984, Cogn. Sci..

[31]  KURT VANLEHN.  Bayesian student modeling, user interfaces and feedback : A sensitivity analysis , 2001 .

[32]  Ding Shuliang,et al.  Test Construction for Cognitive Diagnosis , 2010, 2010 Asia-Pacific Conference on Wearable Computing Systems.

[33]  E. Maris.  Psychometric latent response models , 1995 .

[34]  David J. Spiegelhalter,et al.  Bayesian analysis in expert systems , 1993 .

[35]  Zachary A. Pardos,et al.  The Effect of Model Granularity on Student Performance Prediction Using Bayesian Networks , 2007, User Modeling.

[36]  E. Maris.  Estimating multiple classification latent class models , 1999 .

[37]  Joel D. Martin,et al.  J. Evaluation on an assessment system based on Bayesian student modeling , 1997 .

[38]  R. Siegler.  Three aspects of cognitive development , 1976, Cognitive Psychology.

[39]  Steffen L. Lauritzen,et al.  Bayesian updating in causal probabilistic networks by local computations , 1990 .

[40]  Allan S. Cohen,et al.  Model Selection Methods for Mixture Dichotomous IRT Models , 2009 .

[41]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[42]  Robert J. Mislevy,et al.  PROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS , 1994 .

[43]  M. Reckase.  The Past and Future of Multidimensional Item Response Theory , 1997 .

[44]  Jonathan Templin,et al.  Robustness of Hierarchical Modeling of Skill Association in Cognitive Diagnosis Models , 2008 .

[45]  Jimmy de la Torre,et al.  A Cognitive Diagnosis Model for Cognitively Based Multiple-Choice Options. , 2009 .

[46]  K. VanLehn.  Mind Bugs: The Origins of Procedural Misconceptions , 1990 .

[47]  Raymond J. Adams,et al.  The Multidimensional Random Coefficients Multinomial Logit Model , 1997 .

[48]  Aileen B. Sedman,et al.  A longitudinal study , 1987 .

[49]  L. S. Cox.  Systematic Errors in the Four Vertical Algorithms in Normal and Handicapped Populations. , 1975 .

[50]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[51]  Fumiko Samejima,et al.  Acceleration model in the heterogeneous case of the general graded response model , 1995 .

[52]  Brian W. Junker,et al.  Stochastic ordering using the latent trait and the sum score in polytomous IRT models , 1997 .

[53]  Cristina Conati,et al.  Using Bayesian Networks to Manage Uncertainty in Student Modeling , 2002, User Modeling and User-Adapted Interaction.

[54]  Daniel Zelterman,et al.  Bayesian Artificial Intelligence , 2005, Technometrics.