A Critique of Kodaganallur, Weitz and Rosenthal, "A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms"

Research on intelligent tutoring systems (ITS) has produced many systems, but only a handful of design principles. From its start in the late 1970s, researchers have recognized that design principles should ideally be derived from psychological insights into the cognitive processes underlying the acquisition of cognitive skills, but attempts to base design philosophies on psychological principles are still few and far between (Ohlsson, 1991). Two such philosophies have come to be associated with the labels model-tracing (MT), which is based on the ACT-R theory of human cognition Given two design philosophies, it is useful to conduct systematic comparisons to ascertain the importance and implications of their differences. In a recent IJAIED article, Kodaganallur, Weitz and Rosenthal (2005), henceforth referred to as KWR, undertake a comparison between MT and CBM. They built two tutoring systems for the same domain, one based on MT and one on CBM, and they report their observations and reflections with respect to a catalogue of issues. For every issue but one they find some weakness or potential problem with the CBM approach and also some reason for believing that any corresponding or related problem with MT will not be too difficult to overcome. They conclude that CBM tutors are "less resource intensive" to build than MT tutors, but also that MT has a wider range of application ("an MTT can be built for every domain in which a CBMT can be built, but the reverse doesn't hold") and that "the remediation provided by an MTT will be superior", the more so the more complex the learner's task (p. 141). They offer these conclusions with the purpose of providing "guidance for others interested in building intelligent tutoring systems" (p. 118). ITS researchers should beware of their guidance, because there are several problems with their paper. The most serious is that KWR have multiple misconceptions regarding CBM which led them to make suboptimal choices in the implementation of their CBM system. As a result, several of their conclusions are wrong. Their methodology has multiple flaws. They draw sweeping conclusions about the two ITS paradigms on the basis of a limited system-building effort. The domain model for their CBM tutor contains 43 constraints (p. 128) and the model for their MT tutor contains 76 rules (p. 137).