Dealing with uncertainty in test assembly

The recent development of computer technologies enabled test institutes to improve their process of item selection for test administration by means of automated test assembly (ATA). A general framework for ATA consists in adopting mixed-integer programming models which are commonly intended to be solved by commercial solvers. Those softwares, notwithstanding their success in handling most of the basic ATA problems, are not always able to find solutions for highly constrained and large-sized instances. Moreover, all the model coefficients are assumed to be fixed and known, an hypothesis that is not true for the item information functions which are derived from the estimates of item response theory parameters. These restrictions motivated us to find an alternative way to specify and solve ATA models. First, we suggest an application of the chance-constrained (CC) approach (see Charnes and Cooper, 1959) which allows to maximize the α-quantile (usually smaller than 0.05) of the sampling distribution function of the test information function obtained by bootstrapping the calibration process. Secondly, for solving the ATA models, CC or not, we adapted a stochastic meta-heuristic called simulated annealing (SA) proposed by Goffe (1996). This technique can handle large-scale models and non-linear functions. A reformulation of the model by the Lagrangian relaxation helps to find the most feasible/optimal solution and, thanks to the SA, more than one neighbourhood of the space is explored avoiding to be trapped in a local optimum. Several simulations on ATA problems are performed and the solutions are compared to CPLEX 12.8.0 Optimizer, a benchmark solver in the linear programming field. Finally, a real data application shows the potential of our approach in practical situations. The algorithms are coded in the open-source framework Julia. Two packages written in Julia are released for solving the estimation and assembly problems described in this dissertation.

[1]  L. James,et al.  An item analysis of the Conditional Reasoning Test of Aggression. , 2015, The Journal of applied psychology.

[2]  Alexander Shapiro,et al.  Convex Approximations of Chance Constrained Programs , 2006, SIAM J. Optim..

[3]  F. Samejima.  Weakly parallel tests in latent trait theory with some criticisms of classical test theory , 1977 .

[4]  Li Cai,et al.  Characterizing Sources of Uncertainty in Item Response Theory Scale Scores , 2012, Educational and psychological measurement.

[5]  D. F. Marks,et al.  An introduction , 1988, Experientia.

[6]  C. Glas,et al.  Testing Linear Models for Ability Parameters in Item Response Models , 2005, Multivariate behavioral research.

[7]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[8]  Dimitrios Rizopoulos.  ltm: An R Package for Latent Variable Modeling and Item Response Theory Analyses , 2006 .

[9]  Yongpei Guan,et al.  A Chance-Constrained Two-Stage Stochastic Program for Unit Commitment With Uncertain Wind Power Output , 2012, IEEE Transactions on Power Systems.

[10]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[11]  Angela J. Verschoor,et al.  Genetic Algorithms for Automated Test Assembly , 2007 .

[12]  Melvyn Sim,et al.  Robust discrete optimization and network flows , 2003, Math. Program..

[13]  M. Chavance.  [Jackknife and bootstrap]. , 1992, Revue d'epidemiologie et de sante publique.

[14]  D. Eignor.  The standards for educational and psychological testing. , 2013 .

[15]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[16]  Jae-Chun Ban,et al.  Data Sparseness and On‐Line Pretest Item Calibration‐Scaling Methods in CAT , 2002 .

[17]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[18]  J-P Fox.  Stochastic EM for estimating the parameters of a multilevel IRT model. , 2003, The British journal of mathematical and statistical psychology.

[19]  Willem J. van der Linden,et al.  Linear Models for Optimal Test Design , 2005 .

[20]  R. Varadhan,et al.  Simple and Globally Convergent Methods for Accelerating the Convergence of Any EM Algorithm , 2008 .

[21]  R. Tsutakawa,et al.  The effect of uncertainty of item parameter estimation on ability estimates , 1990 .

[22]  Marshall L. Fisher,et al.  The Lagrangian Relaxation Method for Solving Integer Programming Problems , 2004, Manag. Sci..

[23]  A Look at Psychometrics in the Netherlands. , 1985 .

[24]  John T. Scott,et al.  A Practical Way to Select an Optimum Farm Plan Under Risk , 1972 .

[25]  E. Meijering.  A chronology of interpolation: from ancient astronomy to modern signal and image processing , 2002, Proc. IEEE.

[26]  J. Mckillip,et al.  Fundamentals of item response theory , 1993 .

[27]  P. Krokhmal,et al.  Portfolio optimization with conditional value-at-risk objective and constraints , 2001 .

[28]  Joyce T. Chen.  Quadratic Programming for Least-Cost Feed Formulations under Probabilistic Protein Constraints , 1973 .

[29]  John Lygeros,et al.  On the Road Between Robust Optimization and the Scenario Approach for Chance Constrained Optimization Problems , 2014, IEEE Transactions on Automatic Control.

[30]  Justin David Durfee,et al.  Comparison of open-source linear programming solvers. , 2013 .

[31]  Modeling variability in item parameters in educational measurement , 2005 .

[32]  F. Drasgow.  An Evaluation of Marginal Maximum Likelihood Estimation for the Two-Parameter Logistic Model , 1989 .

[33]  J. Neyman,et al.  Consistent Estimates Based on Partially Consistent Observations , 1948 .

[34]  William L. Goffe.  SIMANN: A Global Optimization Algorithm using Simulated Annealing , 1996 .

[35]  Toby Walsh,et al.  Stochastic Constraint Programming: A Scenario-Based Approach , 2009, Constraints.

[36]  R. Rockafellar,et al.  Conditional Value-at-Risk for General Loss Distributions , 2001 .

[37]  Martha L. Stocking,et al.  A Method for Severely Constrained Item Selection in Adaptive Testing , 1992 .

[38]  Li Cai,et al.  A Comparison of Item Parameter Standard Error Estimation Procedures for Unidimensional and Multidimensional Item Response Theory Modeling , 2014 .

[39]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[40]  Garrett Birkhoff,et al.  Smooth Surface Interpolation , 1960 .

[41]  R. Freund.  THE INTRODUCTION OF RISK INTO A PROGRAMMING MODEL , 1956 .

[42]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[43]  A. Charnes,et al.  Cost Horizons and Certainty Equivalents: An Approach to Stochastic Programming of Heating Oil , 1958 .

[44]  A. Charnes,et al.  Deterministic Equivalents for Optimizing and Satisficing under Chance Constraints , 1963 .

[45]  Ralph,et al.  CONDITIONAL VALUE AT RISK , 1999 .

[46]  F. Lord.  A theory of test scores. , 1952 .

[47]  C. S. Kim,et al.  The Deterministic Equivalents of Chance-Constrained Programming , 1990 .

[48]  Brian W. Junker,et al.  Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses , 1999 .

[49]  Stochastic Approximation Methods for Latent Regression Item Response Models , 2010 .

[50]  Bernard P. Veldkamp.  Application of robust optimization to automated test assembly , 2013, Ann. Oper. Res..

[51]  J. Fox,et al.  Fixed effects IRT model , 2006 .

[52]  C. R. Deboor,et al.  A practical guide to splines , 1978 .

[53]  Harvey J. Greenberg,et al.  State-of-the-Art Decision-Making Tools in the Information-Intensive Age , 2008 .

[54]  B. Veldkamp,et al.  Prior Distributions for Item Parameters in IRT Models , 2012 .

[55]  M. R. Novick.  The axioms and principal results of classical test theory , 1965 .

[56]  Carol M. Woods.  Empirical Histograms in Item Response Theory With Ordinal Data , 2007 .

[57]  Berit Hasler,et al.  Distributional Assumptions in Chance-Constrained Programming Models of Stochastic Water Pollution , 2010 .

[58]  Hua-Hua Chang.  Linear models for optimal test design , 2007 .

[59]  Dimitris Rizopoulos,et al.  ltm: An R Package for Latent Variable Modeling and Item Response Analysis , 2006 .

[60]  R. Mislevy.  Estimating latent distributions , 1984 .

[61]  William F. Egan.  Applications and Extensions , 2008 .

[62]  S. Downing.  Twelve Steps for Effective Test Development. , 2006 .

[63]  Christine M. Anderson-Cook,et al.  Book review: quantitative risk management: concepts, techniques and tools, revised edition, by A.F. McNeil, R. Frey and P. Embrechts. Princeton University Press, 2015, ISBN 978-0-691-16627-8, xix + 700 pp. , 2017, Extremes.

[64]  Claude J. P. Bélisle.  Convergence theorems for a class of simulated annealing algorithms on ℝd , 1992 .

[65]  Jussi Keppo,et al.  Managing electricity market price risk , 2003, Eur. J. Oper. Res..

[66]  R. D. Bock,et al.  Adaptive EAP Estimation of Ability in a Microcomputer Environment , 1982 .

[67]  R. Philip Chalmers,et al.  mirt: A Multidimensional Item Response Theory Package for the R Environment , 2012 .

[68]  Erling B. Andersen,et al.  Sufficient statistics and latent trait models , 1977 .

[69]  R. Rockafellar,et al.  Optimization of conditional value-at risk , 2000 .

[70]  R. D. Bock,et al.  Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm , 1981 .

[71]  Mariagiulia Matteucci,et al.  Uncertainties in the Item Parameter Estimates and Robust Automated Test Assembly , 2013 .