Empirical Bayes and Item-Clustering Effects in a Latent Variable Hierarchical Model

Empirical Bayes regression procedures are often used in educational and psychological testing as extensions to latent variables models. The National Assessment of Educational Progress (NAEP) is an important national survey using such procedures. The NAEP applies empirical Bayes methods to models from item response theory to calibrate student responses to questions of varying difficulty. Due partially to the limited computing technology that existed when NAEP was first conceived, NAEP analyses are carried out using a two-stage estimation procedure that ignores uncertainty about some model parameters. Furthermore, the item response theory model that NAEP uses ignores the effect of item clustering created by the design of a test form. Using Markov chain Monte Carlo, we simultaneously estimate all parameters of an expanded model that considers item clustering to investigate the impact of item clustering and ignoring uncertainty about model parameters on an important outcome measure that NAEP report. Ignoring these two effects causes substantial underestimation of standard errors and induces a modest bias in location estimates.

[1]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[2]  Edward H. Ip,et al.  Locally dependent latent trait model and the dutch identity revisited , 2002 .

[3]  Alan M. Zaslavsky,et al.  A Hierarchical Latent Variable Model for Ordinal Data with \No Answer" Responses , 1997 .

[4]  Scott L. Zeger,et al.  Generalized linear models with random e ects: a Gibbs sampling approach , 1991 .

[5]  Louise Ryan,et al.  Latent Variable Models for Teratogenesis Using Multiple Binary Outcomes , 1997 .

[6]  Robert J. Mislevy,et al.  Estimation of Latent Group Effects , 1985 .

[7]  Andrea Rotnitzky,et al.  Regression Models for Discrete Longitudinal Responses , 1993 .

[8]  Bradley P. Carlin,et al.  Predicting Working Memory Failure: A Subjective Bayesian Approach to Model Selection , 1992 .

[9]  N. Thomas,et al.  Generating Multiple Imputations for Matrix Sampling Data Analyzed With Item Response Models , 1997 .

[10]  Richard J. Patz,et al.  The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data , 2002 .

[11]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[12]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1968 .

[13]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[14]  L. Tierney,et al.  Accurate Approximations for Posterior Moments and Marginal Densities , 1986 .

[15]  A. Shapiro.  Monte Carlo Sampling Methods , 2003 .

[16]  I. M. Pyshik,et al.  Table of integrals, series, and products , 1965 .

[17]  Nancy L. Allen,et al.  The NAEP 1998 Technical Report. , 2001 .

[18]  Jim Albert,et al.  Ordinal Data Modeling , 2000 .

[19]  A. Rukhin.  Bayes and Empirical Bayes Methods for Data Analysis , 1997 .

[20]  Nicholas T. Longford.  Models for Uncertainty in Educational Testing , 1995 .

[21]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[22]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[23]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[24]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[25]  I. S. Gradshteyn,et al.  Table of Integrals, Series, and Products , 1976 .

[26]  Raymond J. Adams,et al.  Multilevel Item Response Models: An Approach to Errors in Variables Regression , 1997 .

[27]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[28]  Robert K. Tsutakawa,et al.  Approximation for Bayesian Ability Estimation , 1988 .

[29]  Francis Tuerlinckx,et al.  The effect of ignoring local item dependencies on the estimated discrimination parameter , 1998 .

[30]  Robert J. Mislevy,et al.  Randomization-based inference about latent variables from complex samples , 1991 .

[31]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[32]  B. Efron.  Empirical Bayes Methods for Combining Likelihoods , 1996 .

[33]  George B. Macready,et al.  Concomitant-Variable Latent-Class Models , 1988 .

[34]  Russell G. Almond,et al.  Graphical Models and Computerized Adaptive Testing , 1998 .

[35]  D. Rubin,et al.  Statistical Analysis with Missing Data , 1988 .

[36]  Edward H. Ip,et al.  Testing for local dependency in dichotomous and polytomous item response models , 2001 .

[37]  D. Rubin.  Multiple imputation for nonresponse in surveys , 1989 .

[38]  L. Zeger,et al.  Efficient Matrix Sampling Instruments for Correlated Latent Traits: Examples from the National Assessment of Educational Progress , 1997 .

[39]  Edward H. Ip,et al.  Adjusting for information inflation due to local dependency in moderately large item clusters , 2000 .

[40]  Eric T. Bradlow,et al.  A hierarchical latent variable model for ordinal data from a customer satisfaction survey with no answer responses , 1999 .

[41]  Bradley P. Carlin,et al.  BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS , 1996, Stat. Comput..

[42]  Cynthia G. Parshall,et al.  Computerized Adaptive Tests , 2002 .

[43]  Eric T. Bradlow,et al.  A Bayesian random effects model for testlets , 1999 .

[44]  E. Muraki.  A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHM , 1992 .

[45]  S. Zeger,et al.  Multivariate Regression Analyses for Categorical Data , 1992 .

[46]  C. Morris,et al.  Hierarchical Poisson Regression Modeling , 1997 .

[47]  Brian W. Junker,et al.  Essential independence and likelihood-based ability estimation for polytomous items , 1991 .

[48]  D. Horvitz,et al.  A Generalization of Sampling Without Replacement from a Finite Universe , 1952 .

[49]  J. Albert.  Bayesian Estimation of Normal Ogive Item Response Curves Using Gibbs Sampling , 1992 .

[50]  D. Hedeker,et al.  A random-effects ordinal regression model for multilevel analysis. , 1994, Biometrics.

[51]  Wing Hung Wong,et al.  Bayesian Analysis in Applications of Hierarchical Models: Issues and Methods , 1996 .

[52]  Richard J. Patz,et al.  A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models , 1999 .

[53]  Rebecca Zwick,et al.  Assessing the Dimensionality of NAEP Reading Data , 1987 .

[54]  Roger A. Sugden,et al.  Multiple Imputation for Nonresponse in Surveys , 1988 .

[55]  R. Kass,et al.  Approximate Bayesian Inference in Conditionally Independent Hierarchical Models (Parametric Empirical Bayes Models) , 1989 .

[56]  Bradley P. Carlin,et al.  Approaches for Empirical Bayes Confidence Intervals , 1990 .

[57]  Scott L. Zeger,et al.  Latent Variable Regression for Multiple Discrete Outcomes , 1997 .

[58]  N T Longford,et al.  Empirical Bayes methods for estimating hospital-specific mortality rates. , 1994, Statistics in medicine.

[59]  Howard Wainer,et al.  Testlet Response Theory: An Analog for the 3PL Model Useful in Testlet-Based Adaptive Testing , 2000 .