On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests

Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polynomial. Monotonic polynomial items can be used with traditional CAT item selection algorithms that use analytical derivatives. We compared these approaches in CAT simulations with a variety of item selection algorithms. Our simulations also varied the features of the calibration and item pool: sample size, the presence of missing data, and the percentage of nonstandard items. In general, the results support the use of semi- and nonparametric item response functions in a CAT.

[1]  Hua-Hua Chang,et al.  A Global Information Approach to Computerized Adaptive Testing , 1996 .

[2]  Bernard W. Silverman,et al.  Temperature schedules for simulated annealing , 1994 .

[3]  David Magis,et al.  Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR , 2012 .

[4]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[5]  E. Muraki.  A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHM , 1992 .

[6]  J. Ramsay.  Kernel smoothing approaches to nonparametric item characteristic curve estimation , 1991 .

[7]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[8]  G. S. Watson,et al.  Smooth regression analysis , 1964 .

[9]  W. M. Yen.  Using Simulation Results to Choose a Latent Trait Model , 1981 .

[10]  E. Nadaraya.  On Estimating Regression , 1964 .

[11]  Antonio Punzo,et al.  KernSmoothIRT: An R Package for Kernel Smoothing in Item Response Theory , 2012, 1211.1183.

[12]  Carl F. Falk.  Model Selection for Monotonic Polynomial Item Response Models , 2017, Springer Proceedings in Mathematics & Statistics.

[13]  Timothy R. Brick,et al.  OpenMx 2.0: Extended Structural Equation and Statistical Modeling , 2015, Psychometrika.

[14]  M. Browne,et al.  A Quasi-Parametric Method for Fitting Flexible Item Response Functions , 2015 .

[15]  Chia-Yi Chiu,et al.  Nonparametric CAT for CD in Educational Settings With Small Samples , 2019, Applied psychological measurement.

[16]  Carl F. Falk,et al.  Maximum Marginal Likelihood Estimation of a Monotonic Polynomial Generalized Partial Credit Model with Applications to Multiple Group Analysis , 2014, Psychometrika.

[17]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[18]  Li Cai,et al.  Semiparametric Item Response Functions in the Context of Guessing. , 2016 .

[19]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[20]  L. Feuerstahler.  Metric Transformations and the Filtered Monotonic Polynomial Item Response Model , 2018, Psychometrika.

[21]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters , 1982 .

[22]  R. D. Bock,et al.  Adaptive EAP Estimation of Ability in a Microcomputer Environment , 1982 .

[23]  Young-Sun Lee,et al.  A Comparison of Methods for Nonparametric Estimation of Item Characteristic Curves for Binary Items , 2007 .

[24]  Willem J. van der Linden,et al.  Bayesian item selection criteria for adaptive testing , 1998 .

[25]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm , 1981 .

[26]  Leah M. Feuerstahler.  Exploring Alternate Latent Trait Metrics with the Filtered Monotonic Polynomial IRT Model , 2016 .

[27]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[28]  Longjuan Liang,et al.  A semi-parametric approach to estimating item response functions , 2007 .

[29]  William Stout,et al.  Nonparametric Item Response Theory: A Maturing and Applicable Measurement Modeling Approach , 2001 .

[30]  Robert J. Mislevy,et al.  Bayes modal estimation in item response models , 1986 .

[31]  Carl F. Falk.  The Monotonic Polynomial Graded Response Model: Implementation and a Comparative Study , 2020, Applied psychological measurement.

[32]  Niels G Waller,et al.  Bayesian Modal Estimation of the Four-Parameter Item Response Model in Real, Realistic, and Idealized Data Sets , 2017, Multivariate behavioral research.

[33]  Jeffrey A Douglas,et al.  Computerized adaptive testing under nonparametric IRT models , 2006 .