Rating ability in performance judgments: The joint influence of implicit theories and intelligence

Abstract The purpose of this study was to approach the issue of rating ability by examining the influence of rater implicit theories and rater intelligence on rating outcomes. Using the inferential accuracy model ( Jackson, 1972 ), raters were identified as either possessing a normative or idiosyncratic implicit theory of the occupation of college instructor. In a laboratory setting, 50 normative and 50 idiosyncratic raters judged the videotaped performance of either a good or poor lecturer. Results showed that (a) intelligence was positively related to rating accuracy and to possessing a normative implicit theory, (b) rater type moderated the relationship between intelligence and rating accuracy, and (c) controlling for intelligence, normative raters committed stronger halo effects than idiosyncratic raters. These results were discussed in relation to furthering the understanding of rating ability.

[1]  Jerard F. Kehoe,et al.  Rater selective attention ability and appraisal effectiveness: The effect of a cognitive style on the accuracy of differentiation among ratees. , 1984 .

[2]  David E. Smith,et al.  Accuracy of performance ratings as affected by rater training and perceived purpose of rating. , 1984 .

[3]  R. Alexander,et al.  The role of inferential accuracy in performance rating. , 1985, Academy of management review. Academy of Management.

[4]  G. Northcraft,et al.  Experts, amateurs, and refrigerators: Comparing expert and amateur negotiators in a novel task , 1986 .

[5]  R. Hastie,et al.  The relationship between memory and judgment depends on whether the judgment task is memory-based or on-line , 1986 .

[6]  D. Jackson,et al.  Clinical judgment of psychopathology: a model for inferential accuracy. , 1975, Journal of Abnormal Psychology.

[7]  Kevin R. Murphy,et al.  Toward Narrowing the Research-Practice Gap in Performance Appraisal. , 1985 .

[8]  D. Jackson,et al.  Decision making in the employment interview: An experimental approach. , 1980 .

[9]  R. Downey,et al.  Rating the ratings: Assessing the psychometric quality of rating data , 1980 .

[10]  Neil M. A. Hauenstein,et al.  FROM LABORATORY TO PRACTICE: NEGLECTED ISSUES IN IMPLEMENTING FRAME‐OF‐REFERENCE RATER TRAINING , 1989 .

[11]  Robert G. Lord,et al.  Cognitive categorization and dimensional schemata: A process approach to the study of halo in performance ratings. , 1983 .

[12]  D. Jackson.  A model for inferential accuracy. , 1972 .

[13]  Milton D. Hakel,et al.  Effects of applicant stereotypes, order, and information on interview impressions. , 1974 .

[14]  S. K. Mitchell.  Interobserver agreement, reliability, and generalizability of data collected in observational studies. , 1979 .

[15]  Walter C. Borman,et al.  Personal constructs, performance schemata, and “folk theories” of subordinate effectiveness: Explorations in an army officer sample☆ , 1987 .

[16]  S. Zedeck,et al.  Development of Behaviorally Anchored Scales for the Evaluation of Faculty Teaching. , 1973 .

[17]  S. Zedeck,et al.  Capturing rater policies for processing evaluation data. , 1977, Organizational behavior and human performance.

[18]  William K. Balzer,et al.  Systematic distortions in memory-based behavior ratings and performance evaluations: Consequences for rating accuracy. , 1986 .

[19]  Kenneth O. Doyle,et al.  Dimensions of Effective Teaching: Factors or Artifacts , 1978 .

[20]  D. Jackson,et al.  Vocational Interests and Personality: An Empirical Integration. , 1970 .

[21]  Angelo S. DeNisi,et al.  A cognitive view of the performance appraisal process: A model and research propositions , 1984 .

[22]  Elaine D. Pulakos,et al.  The development of training programs to increase accuracy with different rating tasks , 1986 .

[23]  Larry E. Mainstone,et al.  An Investigation of the Rater-Ratee Acquaintance and Rater Bias , 1985 .

[24]  D. Funder,et al.  Errors and mistakes: evaluating the accuracy of social judgment. , 1987, Psychological bulletin.

[25]  Michael P. Kirsch,et al.  Job Knowledge, Ratee Familiarity, Conceptual Similarity, and Halo Error: An Exploration. , 1986 .

[26]  Elaine D. Pulakos,et al.  A comparison of rater training programs: Error training and accuracy training. , 1984 .

[27]  Walter C. Borman,et al.  Individual Differences Correlates of Accuracy in Evaluating Others' Performance Effectiveness , 1979 .

[28]  W. Borman,et al.  Implications of Personality Theory and Research for the Rating of Work Performance in Organizations , 1983 .

[29]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[30]  James W. Smither,et al.  True intercorrelation among job components, time delay in rating, and rater intelligence as determinants of accuracy in performance ratings , 1987 .

[31]  L. Cronbach.  Processes affecting scores on understanding of others and assumed similarity. , 1955, Psychological bulletin.

[32]  Kenneth O. Doyle,et al.  Implicit Theories in Student Ratings , 1976 .

[33]  Jack M. Feldman,et al.  Beyond Attribution Theory: Cognitive Processes in Performance Appraisal , 1981 .

[34]  Jack M. Feldman,et al.  Self-generated validity and other effects of measurement on belief, attitude, intention, and behavior. , 1988 .