Simulation Studies for Evaluating the Performance of the Two Classification Methods in the AHM

The attribute hierarchy method (AHM) (Leighton, Gierl, & Hunka, 2004), which is based on the assumption that test items can be described by a set of hierarchically ordered attributes, is designed to estimate examinees’ level of competency as well as their profiles for mastering a set of attributes. The AHM, by incorporating the assumption of attribute dependency, brings an important cognitive property into cognitive diagnostic testing. This study used simulated response vectors to evaluate the performance of the two classification methods (method A and B) for the AHM. The performance of the two classification methods was evaluated at the attribute pattern level and the individual attribute level, respectively. Our results suggest that classification method B outperforms method A at both the attribute pattern and the individual attribute level. In addition, the performance of two methods at the individual attribute level is considerably higher than their performance at the attribute pattern level.

[1]  Michael Scriven.  The Nature of Evaluation. Part I: Relation to Psychology. ERIC/AE Digest. , 1999 .

[2]  Gary Buck,et al.  The testing of listening comprehension: an introspective study1 , 1991 .

[3]  S. Chipman,et al.  Cognitively diagnostic assessment , 1995 .

[4]  Kikumi Tatasuoka.  Use of Generalized Person-Fit Indexes, Zetas for Statistical Pattern Classification. , 1996 .

[5]  Alexandre Gaudeul,et al.  The review of educational research. , .

[6]  K. Tatsuoka,et al.  Application of the rule-space procedure to language testing: examining attributes of a free response listening test , 1998 .

[7]  Paul D. Nichols,et al.  A Framework for Developing Cognitively Diagnostic Assessments , 1994 .

[8]  Mark J. Gierl,et al.  An NCME Instructional Module on Exploring the Logic of Tatsuoka's Rule‐Space Model for Test Development and Analysis , 2005 .

[9]  Robert J. Mislevy,et al.  PROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS , 1994 .

[10]  Ellen B. Mandinach,et al.  INTEGRATING ASSESSMENT AND INSTRUCTION: A RESEARCH AND DEVELOPMENT AGENDA , 1991 .

[11]  Robert J. Mislevy,et al.  BILOG 3 : item analysis and test scoring with binary logistic models , 1990 .

[12]  Kikumi K. Tatsuoka,et al.  Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. , 1995 .

[13]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[14]  Gary Buck,et al.  The appropriacy of psychometric measurement models for testing second language listening comprehension , 1994 .

[15]  K. Tatsuoka.  RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .