TOWARD A TEST THEORY FOR ASSESSING STUDENT UNDERSTANDING

Abstract : The view of learning that underlies standard test theory is inconsistent with the view rapidly emerging from cognitive and educational psychology. Learners become more competent not simply by learning more facts and skills, but by reconfiguring their knowledge; by 'chunking' information to reduce memory loads; and by developing strategies and models that help them discern when and how facts and skills are important. Neither classical test theory nor item response theory (IRT) is designed to inform educational decisions conceived from this perspective. This paper sketches the outlines of a test theory built around models of student understanding, as inspired by the substance and the psychology of the domain of interest. The ideas are illustrated with a simple numerical example based on Siegler's balance beam tasks. Directions in which the approach must be developed to be broadly useful in educational practice are discussed.

[1]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[2]  George B. Macready,et al.  The application of latent class models in adaptive testing , 1992 .

[3]  Robert J. Mislevy,et al.  Integrating Cognitive and Psychometric Models to Measure Document Literacy. , 1990 .

[4]  Kentaro Yamamoto,et al.  A Model That Combines IRT and Latent Class Models , 1987 .

[5]  Paul J. Feltovich,et al.  Categorization and Representation of Physics Problems by Experts and Novices , 1981, Cogn. Sci..

[6]  C. Spearman,et al.  Demonstration of Formulae for True Measurement of Correlation , 1907 .

[7]  Robert Karplus,et al.  Early adolescents' proportional reasoning on ‘rate’ problems , 1983 .

[8]  M. Wilson.  A Comparison of Deterministic and Probabilistic Approaches to Measuring Learning Structures , 1989 .

[9]  R. Snow,et al.  Lecture 2: Toward a Theory of Cognitive Aptitude for Learning from Instruction , 1984 .

[10]  R. Hambleton.  Principles and selected applications of item response theory. , 1989 .

[11]  J. Piaget,et al.  The Growth of Logical Thinking , 1959 .

[12]  William J. Clancey,et al.  Qualitative student models , 1986 .

[13]  Mark Wilson.  Saltus: A psychometric model of discontinuity in cognitive development. , 1989 .

[14]  Some Theoretical Concerns about Applying Latent Trait Models in Educational Testing , 1983 .

[15]  R. Siegler.  Developmental Sequences within and between Concepts. , 1981 .

[16]  T. Kuhn.  The Structure of Scientific Revolutions 2nd edition , 1970 .

[17]  Kevin F. Collis,et al.  Evaluating the Quality of Learning: The SOLO Taxonomy , 1977 .

[18]  James A. Paulson.  Latent Class Representation of Systematic Patterns in Test Responses , 1986 .

[19]  Robert J. Mislevy,et al.  Modeling item responses when different subjects employ different solution strategies , 1990 .

[20]  R. Glaser,et al.  The Future of Testing: A Research Agenda for Cognitive Psychology and Psychometrics. , 1981 .

[21]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[22]  Frederic M. Lord,et al.  Statistical inferences about true scores , 1959 .

[23]  Robert K. Tsutakawa,et al.  Bayesian Ability Estimation via 3PL (Three-Parameter Logistic) with Partially Known Item Parameters , 1988 .

[24]  Robert J. Mislevy,et al.  Randomization-based inference about latent variables from complex samples , 1991 .

[25]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters , 1982 .

[26]  J. Clement.  Students’ preconceptions in introductory mechanics , 1982 .

[27]  L. McDermott.  Research on conceptual understanding in mechanics , 1984 .

[28]  Steen Andreassen,et al.  MUNIN - A Causal Probabilistic Network for Interpretation of Electromyographic Findings , 1987, IJCAI.

[29]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[30]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[31]  Edward Haertel,et al.  An Application of Latent Class Models to Assessment Data , 1984 .

[32]  Marvin Minsky,et al.  A framework for representing knowledge" in the psychology of computer vision , 1975 .

[33]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm , 1981 .

[34]  P. Langley,et al.  Identifying Solution Paths in Cognitive Diagnosis. , 1985 .

[35]  K. Tatsuoka.  Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[36]  M. R. Novick.  The axioms and principal results of classical test theory , 1965 .

[37]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[38]  H. Gulliksen.  Theory of mental tests , 1952 .

[39]  Keam-Claude Falmagne,et al.  A latent trait theory via a stochastic learning theory for a knowledge space , 1989 .