An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales.

This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.

[1]  W C Borman,et al.  Consistency of rating accuracy and rating errors in the judgment of human performance. , 1977, Organizational behavior and human performance.

[2]  H. John Bernardin,et al.  Behavioral expectation scales versus summated scales: A fairer comparison. , 1977 .

[3]  C H COOMBS,et al.  Psychological scaling without a unit of measurement. , 1950, Psychological review.

[4]  J. C. Flanagan.  Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[5]  Joseph L. Zinnes,et al.  Probabilistic, multidimensional unfolding analysis , 1974 .

[6]  Robert Rosenthal,et al.  Judgment Studies: Design, Analysis, and Meta-Analysis , 1987 .

[7]  K. Murphy.  Personnel Selection in Organizations , 1992 .

[8]  Frank J. Landy,et al.  The measurement of work performance : methods, theory and applications / Frank J. Landy, James L. Farr , 1983 .

[9]  W. Borman,et al.  Format and training effects on rating accuracy and rater errors , 1979 .

[10]  Marvin D. Dunnette,et al.  The development and evaluation of behaviorally based rating scales , 1973 .

[11]  A. W. Bendig.  Reliability and the number of rating-scale categories. , 1954 .

[12]  Walter C. Borman,et al.  Job behavior, performance, and effectiveness. , 1991 .

[13]  T. Ll.  Psychophysical analysis. By L. L. Thurstone, 1927. , 1987 .

[14]  Robert J. Wherry,et al.  A Study of Leniency in Two Rating Systems , 1951 .

[15]  Ronald A. Berk,et al.  Performance Assessment: Methods and Applications , 1986 .

[16]  D. Schwab,et al.  Behaviorally Anchored Rating Scales: A Review of the Literature. , 1975 .

[17]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[18]  T. Dickinson,et al.  A comparison of the behaviorally anchored rating and mixed standard scale formats. , 1980 .

[19]  Jeffrey S. Kane,et al.  Performance distribution assessment. , 1986 .

[20]  P. C. Smith,et al.  Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. , 1963 .

[21]  D. Organ.  Organizational citizenship behavior: The good soldier syndrome. , 1988 .

[22]  J. Madden,et al.  Effects of variations in rating scale format on judgment. , 1964 .

[23]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[24]  J. W. Parker,et al.  Rating Scale Content: IV. Predictability of Structured and Unstructured Scales1 , 1959 .

[25]  Gary P. Latham,et al.  Increasing productivity through performance appraisal , 1981 .

[26]  Walter C. Borman,et al.  Investigating the Underlying Structure of the Citizenship Performance Domain , 2000 .

[27]  Edwin E. Ghiselli,et al.  THE MIXED STANDARD SCALE: A NEW RATING SYSTEM , 1972 .

[28]  L. Cronbach.  Processes affecting scores on understanding of others and assumed similarity. , 1955, Psychological bulletin.

[29]  S. J. Motowidlo,et al.  Prosocial Organizational Behaviors , 1986 .

[30]  Wade M. Gibson,et al.  Personnel Selection and Placement , 1988 .

[31]  W. Borman,et al.  Expanding the Criterion Domain to Include Elements of Contextual Performance , 1993 .