Automated Simultaneous Assembly of Multistage Testlets for a High-Stakes Licensing Examination

Many challenges exist for high-stakes testing programs offering continuous computerized administration. The automated assembly of test questions to exactly meet content and other requirements, provide uniformity, and control item exposure can be modeled and solved by mixed-integer programming (MIP) methods. A case study of the computerized licensing examination of the American Institute of Certified Public Accountants is offered as one application of MIP techniques for test assembly. The solution illustrates assembly for a computer-adaptive multistage testing design. However, the general form of the constraint-based solution can be modified to generate optimal test designs for paper-based or computerized administrations, regardless of the specific psychometric model. An extension of this methodology allows for long-term planning for the production and use of test content on the basis of an exact psychometric test designs and administration schedules.

[1]  Martha L. Stocking,et al.  Practical Issues in Large-Scale Computerized Adaptive Testing , 1996 .

[2]  Cornelis A.W. Glas,et al.  Computerized adaptive testing : theory and practice , 2000 .

[3]  Richard M. Luecht,et al.  Test Information Targeting Strategies for Adaptive Multistage Testing Designs. , 2003 .

[4]  Identifiers California,et al.  Annual Meeting of the National Council on Measurement in Education , 1998 .

[5]  M. Stocking,et al.  A Model and Heuristic For Solving Very Large Item Selection Problems , 1993 .

[6]  Jos J. Adema,et al.  Simultaneous assembly of multiple test forms , 1998 .

[7]  T. Theunissen.  Binary programming and test design , 1985 .

[8]  Bernard P. Veldkamp.  Principles and Methods of Constrained Test Assembly , 2001 .

[9]  Bernard P. Veldkamp,et al.  Automated Simultaneous Assembly for Multistage Testing , 2005 .

[10]  Melvin R. Novick,et al.  Some latent train models and their use in inferring an examinee's ability , 1966 .

[11]  R. Hambleton,et al.  An NCME Instructional Module on Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development. , 2005 .

[12]  Richard M. Leucht.  Computer-Assisted Test Assembly Using Optimization Heuristics. , 1998 .

[13]  Martha L. Stocking,et al.  An Alternative Method for Scoring Adaptive Tests , 1996 .

[14]  Richard M. Luecht,et al.  Implementing the Computer-Adaptive Sequential Testing (CAST) Framework To Mass Produce High Quality Computer-Adaptive and Mastery Tests. , 2000 .

[15]  van der Linden,et al.  Constrained adaptive testing with shadow tests , 2000 .

[16]  Donovan R. Hare,et al.  Assembling an inventory of multistage adaptive testing systems , 2009 .

[17]  Mark J. Gierl,et al.  Construction of Automated Parallel Forms and Multiple Parallel Panels in Computer-Adaptive Sequential Testing : New Measures of Parallelism and Their Applications , 2001 .

[18]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[19]  Richard M. Luecht,et al.  Some Practical Examples of Computer‐Adaptive Sequential Testing , 1998 .

[20]  Willem J. van der Linden,et al.  Linear Models for Optimal Test Design , 2005 .