Computerized Adaptive Testing for Polytomous Motivation Items: Administration Mode Effects and a Comparison With Short Forms

In a randomized experiment (n = 515), a computerized and a computerized adaptive test (CAT) are compared. The item pool consists of 24 polytomous motivation items. Although items are carefully selected, calibration data show that Samejima's graded response model did not fit the data optimally. A simulation study is done to assess possible consequences of model misfit. CAT efficiency was studied by a systematic comparison of the CAT with two types of conventional fixed length short forms, which are created to be good CAT competitors. Results showed no essential administration mode effects. Efficiency analyses show that CAT outperformed the short forms in almost all aspects when results are aggregated along the latent trait scale. The real and the simulated data results are very similar, which indicate that the real data results are not affected by model misfit.

[1]  P. Fayers.  Item Response Theory for Psychologists , 2004, Quality of Life Research.

[2]  E. Muraki.  A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHM , 1992 .

[3]  Computerized Adaptive Measurements of Attitudes. , 1990 .

[4]  Z. Ying,et al.  a-Stratified Multistage Computerized Adaptive Testing with b Blocking , 2001 .

[5]  L. Andries van der Ark,et al.  Relationships and Properties of Polytomous Item Response Theory Models , 2001 .

[6]  N. Waller.  Searching for structure in the MMPI , 1999 .

[7]  Klaas Sijtsma,et al.  MPS5 for Windows. A program for Mokken scale analysis for polytomous items , 2000 .

[8]  R. J. Mokken,et al.  A Theory and Procedure of Scale Analysis: With Applications in Political Research , 1971 .

[9]  M. Petersen,et al.  Introduction to Nonparametric Item Response Theory , 2005, Quality of Life Research.

[10]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[11]  M. Reckase.  Unifactor Latent Trait Models Applied to Multifactor Tests: Results and Implications , 1979 .

[12]  S. Reise,et al.  Computerization and Adaptive Administration of the NEO PI-R , 2000, Assessment.

[13]  Robert J. Mokken,et al.  Nonparametric Models for Dichotomous Responses , 1997 .

[14]  Robert Englebretson.  Searching for Structure: The problem of complementation in colloquial Indonesian conversation , 2003 .

[15]  Fritz Drasgow,et al.  Fitting Polytomous Item Response Theory Models to Multiple-Choice Tests , 1995 .

[16]  R. Darrell Bock,et al.  Fitting a response model forn dichotomously scored items , 1970 .

[17]  Deniz S. Ones,et al.  Psychometric equivalence of the computer and booklet forms of the MMPI: A meta-analysis , 1999 .

[18]  D. Andrich.  Application of a Psychometric Rating Model to Ordered Categories Which Are Scored with Successive Integers , 1978 .

[19]  Steven P Reise,et al.  A Discussion of Modern Versus Traditional Psychometrics As Applied to Personality Assessment Scales , 2003, Journal of personality assessment.

[20]  Y. Hochberg.  A sharper Bonferroni procedure for multiple tests of significance , 1988 .

[21]  S Stark,et al.  Fitting Item Response Theory Models to Two Personality Inventories: Issues and Insights , 2001, Multivariate behavioral research.

[22]  R. Hambleton,et al.  Item Response Theory , 1984, The History of Educational Measurement.

[23]  Wagner A. Kamakura,et al.  Measuring Consumer Attitudes toward the Marketplace with Tailored Interviews , 1989 .

[24]  D. Weiss.  Computerized Adaptive Testing for Effective and Efficient Measurement in Counseling and Education , 2004 .

[25]  Scott L. Hershberger,et al.  The New Rules of Measurement : What Every Psychologist and Educator Should Know , 1999 .

[26]  Jagdip Singh,et al.  Adaptive Designs for Likert-Type Data: An Approach for Implementing Marketing Surveys , 1990 .

[27]  F. Samejima.  Graded Response Model , 1997 .

[28]  N G Waller,et al.  Computerized adaptive personality assessment: an illustration with the Absorption scale. , 1989, Journal of personality and social psychology.

[29]  Ivo W. Molenaar,et al.  Nonparametric Models for Polytomous Responses , 1997 .

[30]  Barbara G. Dodd,et al.  Operational Characteristics of Adaptive Testing Procedures Using the Graded Response Model , 1989 .

[31]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[32]  F. Samejima.  Estimation of latent ability using a response pattern of graded scores , 1969 .

[33]  H. Vorst,et al.  A Randomized Experiment to Compare Conventional, Computerized, and Computerized Adaptive Administration of Ordinal Polytomous Attitude Items , 2005 .