A proposed framework for conducting data-based test analysis.

The authors argue that the current state of applied data-based test analytic practice is unstructured and unmethodical due in large part to the fact that there is no clearly specified, widely accepted test analytic framework for judging the performances of particular tests in particular contexts. Drawing from the extant test theory literature, they propose a rationale that may be used in data-based test analysis. The components of the proposed test analytic framework are outlined in detail, as are examples of the framework as applied to commonly encountered test evaluative scenarios. A number of potential extensions of the framework are discussed.

[1]  F. Lord A theory of test scores. , 1952 .

[2]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[3]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[4]  Clifford C. Clogg,et al.  Latent Variables Analysis: Applications for Developmental Research. , 1995 .

[5]  Peter C. M. Molenaar,et al.  The Relationship Between the Structure of Interindividual and Intraindividual Variability: A Theoretical and Empirical Vindication of Developmental Systems Theory , 2003 .

[6]  K. Jöreskog Testing a simple structure hypothesis in factor analysis , 1966, Psychometrika.

[7]  J. Vermunt Latent Class Models , 2004 .

[8]  L. Guttman The Irrelevance of Factor Analysis for the Study of Group Differences. , 1992, Multivariate behavioral research.

[9]  James,et al.  A History of Factor Indeterminacy , 2004 .

[10]  L. Shepard Chapter 9: Evaluating Test Validity , 1993 .

[11]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[12]  Thomas P. Hogan,et al.  An Empirical Study of Reporting Practices Concerning Measurement Validity , 2004 .

[13]  Roderick P. McDonald Factor interaction in nonlinear factor analysis. , 1967 .

[14]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[15]  Ming-Mei Wang,et al.  Some new results on factor indeterminacy , 1972 .

[16]  P. Rosenbaum,et al.  Conditional Association and Unidimensionality in Monotone Latent Variable Models , 1985 .

[17]  Neil Henry Latent structure analysis , 1969 .

[18]  J. Loevinger Objective Tests as Instruments of Psychological Theory , 1957 .

[19]  R. Lennox,et al.  Conventional wisdom on measurement: A structural equation perspective. , 1991 .

[20]  S. Stouffer,et al.  Measurement and Prediction , 1954 .

[21]  D. Grayson,et al.  Two-group classification in latent trait theory: Scores with monotone likelihood ratio , 1988 .

[22]  S. Whitely Construct validity: Construct representation versus nomothetic span. , 1983 .

[23]  S. Hershberger,et al.  A Simple Rule for Generating Equivalent Models in Covariance Structure Modeling. , 1990, Multivariate behavioral research.

[24]  K. Jöreskog A general approach to confirmatory maximum likelihood factor analysis , 1969 .

[25]  Kathleen L. Slaney,et al.  The logic of test analysis: An evaluation of test theory and a proposed logic for test analysis , 2006 .

[26]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[27]  S. Messick Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[28]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[29]  F. Samejima Graded Response Model , 1997 .

[30]  T Raykov,et al.  On Structural Equation Model Equivalence. , 1999, Multivariate behavioral research.

[31]  Clifford C. Clogg,et al.  Handbook of statistical modeling for the social and behavioral sciences , 1995 .

[32]  Louis Guttman,et al.  THE DETERMINACY OF FACTOR SCORE MATRICES WITH IMPLICATIONS FOR FIVE OTHER BASIC PROBLEMS OF COMMON‐FACTOR THEORY1 , 1955 .

[33]  R. P. McDonald,et al.  Test Theory: A Unified Treatment , 1999 .

[34]  John Hattie,et al.  An Empirical Study of Various Indices for Determining Unidimensionality. , 1984, Multivariate behavioral research.

[35]  AN ADDITIVE METRIC FROM ALL THE PRINCIPAL COMPONENTS OF A PERFECT SCALE1 , 1955 .

[36]  Susan R. Davis,et al.  Trends in Reporting Psychometric Properties of Scales Used in Counseling Psychology Research. , 1990 .

[37]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[38]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[39]  A. Beck,et al.  Beck Depression Inventory-II Items Associated With Self-Reported Symptoms of ADHD in Adult Psychiatric Outpatients , 2003, Journal of Personality Assessment.

[40]  TECHNICAL recommendations for psychological tests and diagnostic techniques. , 1954, Psychological bulletin.

[41]  I. Stelzl Changing a Causal Hypothesis without Changing the Fit: some Rules for Generating Equivalent Path Models. , 1986, Multivariate behavioral research.

[42]  S. Messick Test Validity: A Matter of Consequence , 1998 .

[43]  Eiji Muraki,et al.  Fitting a Polytomous Item Response Model to Likert-Type Data , 1990 .

[44]  Educational Evaluation Standards for Educational and Psychological Testing , 1999 .

[45]  Paul F. Lazarsfeld,et al.  Latent Structure Analysis. , 1969 .

[46]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[47]  S. Blinkhorn Past imperfect, future conditional: Fifty years of test theory , 1997 .

[48]  K. Slaney,et al.  An Analysis of Meehl's MAXCOV-HITMAX Procedure for the Case of Dichotomous Indicators , 2003, Multivariate behavioral research.

[49]  Lloyd G. Humphreys,et al.  Intelligence Measurement Theory And Public Policy Proceedings Of A Symposium In Honor Of Lloyd G Humphreys , 1989 .

[50]  David Thissen,et al.  A taxonomy of item response models , 1986 .

[51]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[52]  Yu-Pin Hu,et al.  A Dynamic Factor Model , 2003 .

[53]  R. Lissitz,et al.  Limitations of Coefficient Alpha as an Index of Test Unidimensionality1 , 1977 .

[54]  J. Greenberg,et al.  An Item Response Theory for Personality and Attitude Scales: Item Analysis Using Restricted Factor Analysis , 1983 .

[55]  F. Bookstein,et al.  Two Structural Equation Models: LISREL and PLS Applied to Consumer Exit-Voice Theory , 1982 .

[56]  Howard Wainer,et al.  Estimating Coefficients in Linear Models: It Don't Make No Nevermind , 1976 .

[57]  S. Messick Test validity and the ethics of assessment. , 1980 .

[58]  Ulman Lindenberger,et al.  Understanding Human Development: Dialogues With Lifespan Psychology , 2003 .

[59]  Gideon J. Mellenbergh,et al.  Measurement precision in test score and item response models , 1996 .

[60]  D. Borsboom,et al.  The Theoretical Status of Latent Variables , 2003 .

[61]  H. Blalock Causal Inferences in Nonexperimental Research , 1966 .

[62]  Frederic M. Lord THE RELATION OF TEST SCORE TO THE TRAIT UNDERLYING THE TEST , 1952 .

[63]  P. Barrett Structural equation modelling : Adjudging model fit , 2007 .

[64]  John Hattie,et al.  Methodology Review: Assessing Unidimensionality of Tests and ltenls , 1985 .

[65]  Johanna E. Nilsson,et al.  Practices Regarding Reporting of Reliability Coefficients: A Review of Three Journals , 1999 .

[66]  Alexander von Eye,et al.  Latent Variables Analysis: Applications for Developmental Research. , 1995 .

[67]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[68]  Dale Whhtington,et al.  How Well Do Researchers Report their Measures? an Evaluation of Measurementin Published Educational Research , 1998 .

[69]  Peter C. M. Molenaar,et al.  Dynamic Latent Variable Models in Developmental Psychology , 1994 .

[70]  Peter C M Molenaar,et al.  Statistical Modeling of the Individual: Rationale and Application of Multivariate Stationary Time Series Analysis , 2005, Multivariate behavioral research.

[71]  D. Campbell,et al.  Convergent and discriminant validation by the multitrait-multimethod matrix. , 1959, Psychological bulletin.

[72]  Henk A. L. Kiers,et al.  Why Factor Analysis Often is the Incorrect Model for Analyzing Bipolar Concepts, and What Model to Use Instead , 1994 .

[73]  F. Krauss Latent Structure Analysis , 1980 .

[74]  P. Molenaar,et al.  Relating Factor Models for Longitudinal Data to Quasi-Simplex and NARMA Models , 2005, Multivariate behavioral research.

[75]  Lee J. Cronbach,et al.  Construct validation after thirty years. , 1989 .

[76]  Wm. R. Wright General Intelligence, Objectively Determined and Measured. , 1905 .

[77]  Peter C. M. Molenaar,et al.  A dynamic factor model for the analysis of multivariate time series , 1985 .

[78]  Daniel Katz,et al.  Research Methods in the Behavioral Sciences. , 1954 .

[79]  R. Linn Educational measurement, 3rd ed. , 1989 .

[80]  F. Samejima Estimation of latent ability using a response pattern of graded scores , 1968 .

[81]  Richard Goldstein Latent Class and Discrete Latent Trait Models: Similarities and Differences , 1998 .

[82]  G. J. Mellenbergh,et al.  Generalized linear item response theory. , 1994 .

[83]  Klaas Sijtsma,et al.  New Developments in Categorical Data Analysis for the Social and Behavioral Sciences , 2005 .

[84]  R. Chrisjohn,et al.  CA and SPOD for the Analysis of Tests Comprised of Binary Items , 1998 .

[85]  E. J. Burr,et al.  A comparison of four methods of constructing factor scores , 1967 .

[86]  P. Holland On the sampling theory roundations of item response theory models , 1990 .

[87]  R. Bagozzi,et al.  On the nature and direction of relationships between constructs and measures. , 2000, Psychological methods.