Evaluation parameters for computer-adaptive testing

With the proliferation of computers in test delivery today, adaptive testing has become quite popular, especially when examinees must be classified into two categories (pass/fail, master/nonmaster). Several well-established organisations have provided standards and guidelines for the design and evaluation of educational and psychological testing. The purpose of this paper was not to repeat the guidelines and standards that exist in the literature but to identify and discuss the main evaluation parameters for a computer-adaptive test (CAT). A number of parameters should be taken into account when evaluating CAT. Key parameters include utility, validity, reliability, satisfaction, usability, reporting, administration, security, and thoseassociated with adaptivity, item pool, and psychometric theory. These parameters are presented and discussed below and form a proposed evaluation model, Evaluation Model of Computer-Adaptive Testing.

[1]  The International Test Commission.  International Guidelines on Computer-Based and Internet-Delivered Testing , 2006 .

[2]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[3]  Peter Brusilovsky,et al.  Adaptive Hypermedia , 2001, User Modeling and User-Adapted Interaction.

[4]  Peter Brusilovsky,et al.  Methods and techniques of adaptive hypermedia , 1996, User Modeling and User-Adapted Interaction.

[5]  Andreas S. Pomportsis,et al.  The value of adaptivity based on cognitive style: an empirical study , 2004, Br. J. Educ. Technol..

[6]  “State-of-the-Art and Adaptive Open-Closed Items in Adaptive Foreign Language Assessment " , 2004 .

[7]  Judy Kay.  Lies , damned lies and stereotypes : pragmatic approximations of users , 2003 .

[8]  M. Reckase.  Item pool design for computerized adaptive tests , 2003 .

[9]  Clifford Nass,et al.  Adaptive testing: effects on user performance , 2002, CHI.

[10]  A. Pomportsis,et al.  AES-CS: Adaptive Educational System based on Cognitive Styles , 2002 .

[11]  Mariana Lilley,et al.  The development and evaluation of a computer-adaptive testing application for English language , 2002 .

[12]  George D. Magoulas,et al.  INSPIRE: An INtelligent System for Personalized Instruction in a Remote Environment , 2001, OHS-7/SC-3/AH-3.

[13]  Peter Brusilovsky,et al.  User modeling and user adapted interaction , 2001 .

[14]  Theodorus Johannes Hendrikus Maria Eggen,et al.  Computerized Adaptive Testing for Classifying Examinees into three Categories , 2000 .

[15]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[16]  H. Wainer.  Computerized adaptive testing: A primer, 2nd ed. , 2000 .

[17]  G. Gage Kingsbury,et al.  Practical issues in developing and maintaining acomputerized adaptive testing program , 2000 .

[18]  Patricia A. Dunkel.  Considerations in Developing and Using Computer-Adaptive Tests To Assess Second Language Proficiency. ERIC Digest. , 1999 .

[19]  P.M.E. De Bra,et al.  Design issues in adaptive web-site development , 1999 .

[20]  Anastasios A. Economides,et al.  Evaluation And Comparison Of Web-Based Testing Tools , 1999, WebNet.

[21]  Patricia A. Dunkel,et al.  Considerations in Developing or Using Second/Foreign Language Proficiency Computer-Adaptive Tests , 1999 .

[22]  Udo W. Pooch,et al.  Third Generation Adaptive Hypermedia Systems , 1999, WebNet.

[23]  Daniel R. Eignor,et al.  Guidelines for Computerized-Adaptive Test Development and Use in Education [Book Review]. , 1997 .

[24]  Schloss Birlinghoven,et al.  Adaptability and Adaptivity in Learning Systems , 1997 .

[25]  G. Scott Owen,et al.  Practical issues in multimedia user interface design for computer-based instruction , 1996 .

[26]  David J. Ayersman,et al.  Individual differences, computers, and instruction , 1995 .

[27]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[28]  Lawrence M. Rudner.  Questions To Ask When Evaluating Tests. ERIC/AE Digest. , 1994 .

[29]  Alan Clarke.  The principles of screen design for computer-based learning materials. , 1994 .

[30]  Lawrence M. Rudner.  Questions To Ask When Evaluating Tests. , 1994 .

[31]  J. Mckillip,et al.  Fundamentals of item response theory , 1993 .

[32]  Kevin Cox,et al.  User-interface design (2nd ed.) , 1992 .

[33]  Kathleen M. Sheehan,et al.  Using Bayesian Decision Theory to Design a Computerized Mastery Test , 1990 .

[34]  Harold W. Thimbleby,et al.  User interface design , 1990, ACM Press Frontier Series.

[35]  Jakob Nielsen,et al.  Improving a human-computer dialogue , 1990, CACM.

[36]  D. Weiss.  Adaptive testing by computer. , 1985, Journal of consulting and clinical psychology.

[37]  Mark D. Reckase,et al.  TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS , 1984 .

[38]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .