The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed.

[1]  Kevin R. Murphy,et al.  INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS , 2000 .

[2]  Chockalingam Viswesvaran,et al.  RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY , 2000 .

[3]  John E. Hunter,et al.  Theory Testing and Measurement Error. , 1999 .

[4]  Gordon W. Cheung.  Multifaceted conceptions of self–other ratings disagreement. , 1999 .

[5]  Francis J. Yammarino,et al.  SELF‐OTHER AGREEMENT: DOES IT REALLY MATTER? , 1998 .

[6]  Sarah A. Hezlett,et al.  TRAIT, RATER AND LEVEL EFFECTS IN 360‐DEGREE PERFORMANCE RATINGS , 1998 .

[7]  L. V. Dyne,et al.  Helping and Voice Extra-Role Behaviors: Evidence of Construct and Predictive Validity , 1998 .

[8]  A. H. Church,et al.  Advancing the State of the Art of 360-Degree Feedback , 1997 .

[9]  J. Salgado,et al.  Meta-Analysis of Interrater Reliability of Job Performance Ratings in Validity Studies of Personnel Selection , 1996 .

[10]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[11]  Robert Hooijberg,et al.  A Multidirectional Approach Toward Leadership: An Extension of the Concept of Behavioral Complexity , 1996 .

[12]  F. Schmidt,et al.  Measurement Error in Psychological Research: Lessons From 26 Research Scenarios , 1996 .

[13]  James W. Smither,et al.  CAN MULTI-SOURCE FEEDBACK CHANGE PERCEPTIONS OF GOAL ACCOMPLISHMENT, SELF-EVALUATIONS, AND PERFORMANCE-RELATED OUTCOMES? THEORY-BASED APPLICATIONS AND DIRECTIONS FOR RESEARCH , 1995 .

[14]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[15]  R. Peterson.  A Meta-analysis of Cronbach's Coefficient Alpha , 1994 .

[16]  John P. Campbell,et al.  A Confirmatory Test of a Model of Performance Determinants , 1994 .

[17]  Robert J. Vance,et al.  Construct Validity of Three Types of Organizational Citizenship behavior: An Illustration of the Direct Product Model with Refinements , 1993 .

[18]  Ronald B. Morgan.  SELF-AND CO-WORKER PERCEPTIONS OF ETHICS AND THEIR RELATIONSHIPS TO LEADERSHIP AND SALARY , 1993 .

[19]  P. Muchinsky,et al.  The Validity of the Employment Interview: A Meta‐Analysis , 1993 .

[20]  Jose M. Cortina,et al.  THE “BIG FIVE” PERSONALITY FACTORS IN THE IPI AND MMPI: PREDICTORS OF POLICE PERFORMANCE , 1992 .

[21]  David A. Kravitz,et al.  Context effects in performance appraisal: A methodological critique and empirical study. , 1992 .

[22]  Steven E. Devlin,et al.  Predictive validity of biodata items generated from retrospective life experience essays. , 1990 .

[23]  John R. Weitzel,et al.  A TEST OF THE SITUATIONAL LEADERSHIP THEORY , 1990 .

[24]  Hannah R. Rothstein,et al.  Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. , 1990 .

[25]  John P. Campbell,et al.  Modeling the performance prediction problem in industrial and organizational psychology. , 1990 .

[26]  Manuel London,et al.  RATINGS OF MANAGERIAL CHARACTERISTICS: EVALUATION DIFFICULTY, CO‐WORKER AGREEMENT, AND SELF‐AWARENESS , 1989 .

[27]  Barbara A. Spencer,et al.  Development and Test of a Multivariate Model of Absenteeism , 1989 .

[28]  Neil M. A. Hauenstein,et al.  FROM LABORATORY TO PRACTICE: NEGLECTED ISSUES IN IMPLEMENTING FRAME‐OF‐REFERENCE RATER TRAINING , 1989 .

[29]  Thomas E. Becker,et al.  A FIELD STUDY OF THE RELATIONSHIP BETWEEN THE ORGANIZATIONAL FEEDBACK ENVIRONMENT AND PERFORMANCE , 1989 .

[30]  Kevin R. Murphy,et al.  Multiple uses of performance appraisal: Prevalence and correlates. , 1989 .

[31]  Michael P. Kirsch,et al.  The Effect of Similarity on Performance Ratings and Interrater Agreement , 1989 .

[32]  F. Landy.  Psychology of work behavior, 4th ed. , 1989 .

[33]  Anne S. Tsui,et al.  MULTIPLE ASSESSMENT OF MANAGERIAL EFFECTIVENESS: INTERRATER AGREEMENT AND CONSENSUS IN EFFECTIVENESS MODELS , 1988 .

[34]  John Schaubroeck,et al.  A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. , 1988 .

[35]  Thomas M. Stutzman,et al.  AN EVALUATION OF METHODS TO SELECT RESPONDENTS TO STRUCTURED JOB‐ANALYSIS QUESTIONNAIRES , 1986 .

[36]  Gilbert A. Churchill,et al.  Research Design Effects on the Reliability of Rating Scales: A Meta-Analysis: , 1984 .

[37]  Angelo S. DeNisi,et al.  A cognitive view of the performance appraisal process: A model and research propositions , 1984 .

[38]  H. John Bernardin,et al.  Performance appraisal : assessing human behavior at work , 1984 .

[39]  Laurence Siegel,et al.  PAIRED COMPARISON EVALUATIONS OF MANAGERIAL EFFECTIVENESS BY PEERS AND SUPERVISORS , 1982 .

[40]  Robert J. Wherry,et al.  THE CONTROL OF BIAS IN RATINGS: A THEORY OF RATING , 1982 .

[41]  Stephen G. West,et al.  Validity of self-evaluation of ability: A review and meta-analysis , 1982 .

[42]  Career roles, psychological success, and job attitudes. , 1981 .

[43]  Kevin G. Love.  Comparison of peer assessment methods: Reliability, validity, friendship bias, and user reaction. , 1981 .

[44]  Stephen A. Stumpf,et al.  Career stage as a moderator of performance relationships with facets of job satisfaction and role perceptions , 1981 .

[45]  John E. Hunter,et al.  Halo in a multidimensional forced-choice performance evaluation scale. , 1980 .

[46]  R. Downey,et al.  Rating the ratings: Assessing the psychometric quality of rating data , 1980 .

[47]  W. Borman,et al.  Format and training effects on rating accuracy and rater errors , 1979 .

[48]  Robert L. Holzbach,et al.  Rater bias in performance ratings: Superior, self-, and peer ratings. , 1978 .

[49]  J. Pfeffer,et al.  A social information processing approach to job attitudes and task design. , 1978, Administrative science quarterly.

[50]  Walter C. Borman,et al.  Exploring upper limits of reliability and validity in job performance ratings. , 1978 .

[51]  Wayne F. Cascio,et al.  Applied psychology in personnel management , 1978 .

[52]  A. Bandura.  Social learning theory , 1977 .

[53]  Frank J. Landy,et al.  Psychology of work behavior , 1976 .

[54]  Walter C. Borman,et al.  The rating of individuals in organizations: An alternate approach , 1974 .

[55]  L. Christensen.  The Influence of Trait, Sex, and Information on Accuracy of Personality Assessment , 1974 .

[56]  E. C. Bonk,et al.  MMPI Characteristics of Clergymen in Counseling Training and Their Relationship to Supervisor's and Peers' Ratings of Counseling Effectiveness , 1973, Psychological reports.

[57]  T. Dickinson,et al.  A multitrait-multi method analysis of scales developed by retranslation , 1973 .

[58]  E. Lawler,et al.  Expectancy theory and job behavior , 1973 .

[59]  H. Aurbach,et al.  PREDICTORS, CRITERIA, AND SIGNIFICANT RESULTS , 1971 .

[60]  D. D. Robinson.  PREDICTING POLICE EFFECTIVENESS FROM SELF REPORTS OF RELATIVE TIME SPENT IN TASK PERFORMANCE1 , 1970 .

[61]  E. Lawler,et al.  The multitrait-multirater approach to measuring managerial job performance. , 1967, The Journal of applied psychology.

[62]  V B Cline,et al.  Prediction of creativity and other performance measures from biographical information among pharmaceutical scientists. , 1967, The Journal of applied psychology.

[63]  Paul D. Nelson,et al.  CRITERION MEASURES FOR EXTREMELY ISOLATED GROUPS , 1966 .

[64]  Hollander Ep.  Validity of peer nominations in predicting a distant performance criterion. , 1965 .

[65]  E. Glaser,et al.  Validation of a multiple-assessment procedure for managerial personnel. , 1964 .

[66]  W. J. Mcnamara,et al.  A MANAGERIAL SELECTION STUDY , 1964 .

[67]  Thomas B. Sprecher,et al.  A study of engineers' criteria for creativity. , 1959 .

[68]  A. J. Kubany.  Use of sociometric peer nominations in medical education research. , 1957 .

[69]  H. Strupp,et al.  Non‐Technical Factors in Supervisors' Ratings of Job Performance , 1955 .

[70]  D. L. Grant.  Validity Information Exchange , 1955 .

[71]  D. Springer.  Ratings of candidates for promotion by co-workers and supervisors. , 1953 .

[72]  C. H. Lawshe,et al.  Principles of personnel testing , 1949 .