Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings.

The purpose of this investigation was to examine the psychometric properties (interrater reliabilities within source and correlations between sources) of subordinate, supervisor, peer, and self-ratings of job performance. Different job types and dimension types were compared. Using meta-analytic methodology, we found that subordinates showed the lowest mean reliability (.30) and supervisors showed the highest (.50), with peers in between (.37). Mean correlations between sources were low for subordinate ratings (.22 with supervisor, .22 with peer, and .14 with self-ratings) and for self-ratings (.22 with supervisor and .19 with peer ratings). The mean supervisor-peer correlation was higher at .34. Both reliabilities and correlations between sources tended to be higher for nonmanagerial and lower complexity jobs. Comparisons of between-source correlations with within-source reliabilities indicated that, with some qualifications, the different sources had somewhat different perspectives on performance. Dimen...

[1]  Chockalingam Viswesvaran,et al.  Role of social desirability in personality testing for personnel selection: The red herring. , 1996 .

[2]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[3]  John L. Holland,et al.  Exploring careers with a typology: What we have learned and some new directions. , 1996 .

[4]  John W. Fleenor.  Constructs and developmental assessment centers: Further troubling empirical findings , 1996 .

[5]  James W. Smither,et al.  CAN MULTI-SOURCE FEEDBACK CHANGE PERCEPTIONS OF GOAL ACCOMPLISHMENT, SELF-EVALUATIONS, AND PERFORMANCE-RELATED OUTCOMES? THEORY-BASED APPLICATIONS AND DIRECTIONS FOR RESEARCH , 1995 .

[6]  William H. Bommer,et al.  ON THE INTERCHANGEABILITY OF OBJECTIVE AND SUBJECTIVE MEASURES OF EMPLOYEE PERFORMANCE: A META-ANALYSIS , 1995 .

[7]  Neal Schmitt,et al.  Experience-based and situational interview questions: Studies of validity. , 1995 .

[8]  James W. Smither,et al.  A Field Study of Reactions to Normative versus Individualized Upward Feedback , 1995 .

[9]  David E. Smith,et al.  A FIELD STUDY OF PERFORMANCE APPRAISAL PURPOSE: RESEARCH‐ VERSUS ADMINISTRATIVE‐ BASED RATINGS , 1995 .

[10]  Richard R. Reilly,et al.  AN EXAMINATION OF THE EFFECTS OF AN UPWARD FEEDBACK PROGRAM OVER TIME , 1995 .

[11]  L. Atwater,et al.  THE INFLUENCE OF UPWARD FEEDBACK ON SELF‐ AND FOLLOWER RATINGS OF LEADERSHIP , 1995 .

[12]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[13]  Alan J. Dubinsky,et al.  Transformational leadership theory: Using levels of analysis to determine boundary conditions. , 1994 .

[14]  M. R. Houck,et al.  Tools for assessing situational awareness in an operational fighter environment. , 1994, Aviation, space, and environmental medicine.

[15]  Adrian Furnham,et al.  Congruence of self and subordinate ratings of managerial practices as a correlate of supervisor evaluation , 1994 .

[16]  R. Ayman,et al.  Measuring Leader Behavior: Moderators of Discrepant Self and Subordinate Descriptions , 1994 .

[17]  Subordinates rating managers: Organizational and demographic correlates of self/subordinate agreement , 1993 .

[18]  R M Rose,et al.  Assessments of astronaut effectiveness. , 1993, Aviation, space, and environmental medicine.

[19]  Walter W. Tornow,et al.  Perceptions or reality: Is multi-perspective measurement a means or an end? , 1993 .

[20]  H. J. Bernardin,et al.  Attitudes of first-line supervisors toward subordinate appraisals , 1993 .

[21]  Sarah A. Hezlett,et al.  The impact of 360‐degree feedback on management skills development , 1993 .

[22]  Robert J. Vance,et al.  Construct Validity of Three Types of Organizational Citizenship behavior: An Illustration of the Direct Product Model with Refinements , 1993 .

[23]  Walter C. Borman,et al.  More Progress Toward a Taxonomy of Managerial Performance Requirements , 1993 .

[24]  Ronald B. Morgan.  SELF-AND CO-WORKER PERCEPTIONS OF ETHICS AND THEIR RELATIONSHIPS TO LEADERSHIP AND SALARY , 1993 .

[25]  L. Shore,et al.  Self- and supervisory perspectives on age and work attitudes and performance. , 1992, The Journal of applied psychology.

[26]  Ronald E. Riggio,et al.  Agreement between subordinate and superior ratings of supervisory performance and effects on self and subordinate job satisfaction , 1992 .

[27]  Sue Margaret Norton,et al.  Peer assessments of performance and ability: An exploratory meta-analysis of statistical artifacts and contextual moderators , 1992 .

[28]  Francis J. Yammarino,et al.  DOES SELF‐OTHER AGREEMENT ON LEADERSHIP PERCEPTIONS MODERATE THE VALIDITY OF LEADERSHIP AND PERFORMANCE PREDICTIONS? , 1992 .

[29]  Linda M. Bleicken,et al.  Effects of Supervisor Age and Subordinate Age on Rating Congruence , 1991 .

[30]  Bor-Shiuan Cheng,et al.  Cultural Relativity in Action: A Comparison of Self-Ratings Made by Chinese and U.S. Workers , 1991 .

[31]  Walter C. Borman,et al.  Job behavior, performance, and effectiveness. , 1991 .

[32]  P. Villanova,et al.  Work behavior correlates of interviewer job compatibility , 1990 .

[33]  Steven E. Devlin,et al.  Predictive validity of biodata items generated from retrospective life experience essays. , 1990 .

[34]  Hannah R. Rothstein,et al.  Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. , 1990 .

[35]  John F. Kremer,et al.  Construct Validity of Multiple Measures in Teaching, Research, and Service and Reliability of Peer Ratings. , 1990 .

[36]  John E. Hunter,et al.  Individual differences in output variability as a function of job complexity. , 1990 .

[37]  Shalom Malka,et al.  Application of the multitrait-multirater approach to performance appraisal in social service organizations , 1990 .

[38]  Manuel London,et al.  RATINGS OF MANAGERIAL CHARACTERISTICS: EVALUATION DIFFICULTY, CO‐WORKER AGREEMENT, AND SELF‐AWARENESS , 1989 .

[39]  Glenn M. McEvoy,et al.  Assessment centers and subordinate appraisals of managers: A seven-year examination of predictive validity. , 1989 .

[40]  Donald G. Gardner,et al.  Focus of attention at work: Construct definition and empirical validation , 1989 .

[41]  D. Day,et al.  Personality and job performance: Evidence of incremental validity. , 1989 .

[42]  Michael P. Kirsch,et al.  The Effect of Similarity on Performance Ratings and Interrater Agreement , 1989 .

[43]  J. Hartigan,et al.  Fairness in Employment Testing: Validity Generalization, Minority Issues, and the General Aptitude Test Battery , 1989 .

[44]  Anne S. Tsui,et al.  MULTIPLE ASSESSMENT OF MANAGERIAL EFFECTIVENESS: INTERRATER AGREEMENT AND CONSENSUS IN EFFECTIVENESS MODELS , 1988 .

[45]  John B. Miner.  Development and application of the rated ranking technique in performance appraisal , 1988 .

[46]  John Schaubroeck,et al.  A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. , 1988 .

[47]  B. Bass,et al.  Superiors' evaluations and subordinates' perceptions of transformational and transactional leadership. , 1988 .

[48]  Robert L. Heneman.  The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. , 1986 .

[49]  John E. Hunter,et al.  THE ECONOMIC IMPACT OF JOB SELECTION METHODS ON SIZE, PRODUCTIVITY, AND PAYROLL COSTS OF THE FEDERAL WORK FORCE: AN EMPIRICALLY BASED DEMONSTRATION , 1986 .

[50]  G. R. Ferris,et al.  THE INFLUENCE OF SUBORDINATE AGE ON PERFORMANCE RATINGS AND CAUSAL ATTRIBUTIONS , 1985 .

[51]  James H Harris,et al.  Improving the Selection, Classification, and Utilization of Army Enlisted Personnel. Annual Report, 1984 Fiscal Year , 1985 .

[52]  Nestor K. Ovalle,et al.  SELF‐APPRAISAL BASED UPON SUPERVISORY FEEDBACK , 1984 .

[53]  M. Mount,et al.  Psychometric properties of subordinate ratings of managerial performance. , 1984 .

[54]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[55]  Neal Schmitt,et al.  Validity of assessment center ratings for the prediction of performance ratings and school climate of school administrators. , 1984 .

[56]  J Hogan,et al.  How to measure service orientation. , 1984, The Journal of applied psychology.

[57]  H. Sypher,et al.  SEEING OURSELVES AS OTHERS SEE US , 1984 .

[58]  J. P. Near,et al.  ORGANIZATIONAL CITIZENSHIP BEHAVIOR: ITS NATURE AND ANTECEDENTS , 1983 .

[59]  Joseph G. Rosse,et al.  Reconsidering the vertical dyad linkage model of leadership. , 1983 .

[60]  M. Kaas,et al.  Assessing the Job Performance of Nursing Home Staff , 1983, Nursing research.

[61]  Frank J. Landy,et al.  The measurement of work performance : methods, theory and applications / Frank J. Landy, James L. Farr , 1983 .

[62]  Stephen G. West,et al.  Validity of self-evaluation of ability: A review and meta-analysis , 1982 .

[63]  George Rosinger,et al.  Development of a Behaviorally Based Performance Appraisal System. , 1982 .

[64]  A. Wheeler,et al.  Self, Teacher, and Faculty Assessments of Student Teaching Performance. , 1982 .

[65]  S. J. Motowidlo,et al.  Relationship between self-rated performance and pay satisfaction among sales representatives. , 1982 .

[66]  S. Carroll,et al.  Performance appraisal and review systems : the identification, measurement, and development of performance in organizations , 1982 .

[67]  Bernard M. Bass,et al.  Stogdill's Handbook of Leadership: A Survey of Theory and Research , 1982 .

[68]  Kevin G. Love.  Comparison of peer assessment methods: Reliability, validity, friendship bias, and user reaction. , 1981 .

[69]  John A. Johnson,et al.  VOCATIONAL INTERESTS, PERSONALITY AND EFFECTIVE POLICE PERFORMANCE , 1981 .

[70]  G. M. Worbois.  Validation of Externally Developed Assessment Procedures for Identification of Supervisory Potential , 1975, IEEE Engineering Management Review.

[71]  Raymond Lee,et al.  Multitrait-multimethod-multirater analysis of performance ratings for law enforcement personnel. , 1981 .

[72]  John E. Hunter,et al.  Halo in a multidimensional forced-choice performance evaluation scale. , 1980 .

[73]  L. Baird.  Self and Superior Ratings of Performance: As Related to Self-Esteem and Satisfaction with Supervision , 1977, IEEE Engineering Management Review.

[74]  Cynthia M. Pavett,et al.  The prediction of effort and performance among hospital professionals: Moderating effects of feedback on expectancy theory formulations , 1979 .

[75]  Robert S. Billings,et al.  Comparison of training methods for improving the psychometric quality of experimental and administrative performance ratings. , 1979 .

[76]  Charles N. Greene,et al.  Questions of Causation in the Path-Goal Theory of Leadership1 , 1979 .

[77]  Robert L. Holzbach,et al.  Rater bias in performance ratings: Superior, self-, and peer ratings. , 1978 .

[78]  B. B. Saari,et al.  Behavior, Situation, And Rater Variance In Descriptions Of Leader Behaviors. , 1978, Multivariate behavioral research.

[79]  Jeffrey S. Kane,et al.  Methods of peer assessment. , 1978 .

[80]  J. J. Woodmansee.  Validation of the Nurturance Scale of the Edwards Personal Preference Schedule , 1978, Psychological reports.

[81]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[82]  Ramon J. Aldag,et al.  Moderators of the relationships between self and superior evaluations of job performance , 1977 .

[83]  Frank J. Landy,et al.  The mixed standard rating scale: An evaluation , 1977 .

[84]  Michael J. Stahl,et al.  Improving R&D Productivity: Measuring Innovation and Productivity — A Peer Rating Approach , 1977 .

[85]  Greg R. Oldham,et al.  The motivational strategies used by supervisors: Relationships to effectiveness indicators , 1976 .

[86]  F. Landy,et al.  Behaviorally anchored scales for rating the performance of police officers. , 1976 .

[87]  Bernard M. Bass,et al.  Management styles associated with organizational, task, personal, and interpersonal contingencies. , 1975 .

[88]  R D Griffiths,et al.  The accuracy and correlates of psychiatric patients' self-assessment of their work behaviour. , 1975, The British journal of social and clinical psychology.

[89]  H. G. Heneman,et al.  Comparisons of self- and superior ratings of managerial performance. , 1974 .

[90]  Walter C. Borman,et al.  The rating of individuals in organizations: An alternate approach , 1974 .

[91]  Richard J. Klimoski,et al.  Role of the rater in performance appraisal. , 1974 .

[92]  W. C. Borman,et al.  A View of What Can Happen When Behavioral Expectation Scales are Developed in One Setting and Used in Another , 1974 .

[93]  R D Griffiths,et al.  A Standardized Assessment of the Work Behaviour of Psychiatric Patients , 1973, British Journal of Psychiatry.

[94]  Charles N. Greene,et al.  A LONGITUDINAL ANALYSIS OF RELATIONSHIPS AMONG LEADER BEHAVIOR AND SUBORDINATE PERFORMANCE AND SATISFACTION. , 1973 .

[95]  Charles N. Greene,et al.  Causal connections among managers' merit pay, job satisfaction, and performance. , 1973 .

[96]  John W. Slocum,et al.  Organizational Climate and the Effectiveness of a Human Relations Training Program , 1973 .

[97]  T. Dickinson,et al.  A multitrait-multi method analysis of scales developed by retranslation , 1973 .

[98]  A. R. Bass,et al.  Ethnic group differences in relationships among criteria of job performance. , 1973 .

[99]  Marvin D. Dunnette,et al.  The development and evaluation of behaviorally based rating scales , 1973 .

[100]  W. E. Williams,et al.  Relationship between measures of effort and job performance. , 1973 .

[101]  M. Evans.  LEADERSHIP BEHAVIOR: DEMOGRAPHIC FACTORS AND AGREEMENT BETWEEN SUBORDINATE AND SELF-DESCRIPTIONS , 1972 .

[102]  M. Howell,et al.  Rating the clinical effectiveness of employed physicians. , 1972, The Journal of applied psychology.

[103]  David H. Ryman,et al.  Convergent and discriminant validities of performance evaluations in extremely isolated groups. , 1971 .

[104]  R. Cummins.  Relationship of initiating structure and job performance as moderated by consideration. , 1971, The Journal of applied psychology.

[105]  J. Hackman,et al.  Employee reactions to job characteristics. , 1971 .

[106]  Michael J. Kavanagh,et al.  Issues in managerial performance: Multitrait-multimethod analyses of ratings. , 1971 .

[107]  H. A. Thomson.  Comparison of predictor and criterion judgments of managerial performance using the multitrait-multimethod approach. , 1970 .

[108]  R. Boruch,et al.  Alternative Methods of Analysis: Multitrait-Multimethod Data , 1970 .

[109]  PSYCHOLOGICAL EVALUATIONS AS PREDICTORS OF CONSULTING SUCCESS , 1970 .

[110]  D. D. Robinson.  PREDICTING POLICE EFFECTIVENESS FROM SELF REPORTS OF RELATIVE TIME SPENT IN TASK PERFORMANCE1 , 1970 .

[111]  H. Levine,et al.  The Validity of Multiple Choice Achievement Tests as Measures of Competence in Medicine1 , 1970 .

[112]  F. Landy,et al.  Development of scales for the measurement of work motivation , 1970 .

[113]  George C. Thornton,et al.  THE RELATIONSHIP BETWEEN SUPERVISORY- AND SELF-APPRAISALS OF EXECUTIVE PERFORMANCE , 1968 .

[114]  Edward E. Lawler,et al.  A correlational-causal analysis of the relationship between expectancy attitudes and job performance. , 1968 .

[115]  Lyman W. Porter,et al.  Expectancy theory predictions of work effectiveness. , 1968 .

[116]  E. Lawler,et al.  The multitrait-multirater approach to measuring managerial job performance. , 1967, The Journal of applied psychology.

[117]  V B Cline,et al.  Prediction of creativity and other performance measures from biographical information among pharmaceutical scientists. , 1967, The Journal of applied psychology.

[118]  Roy Payne,et al.  EXTRAVERSION AND SOME ASPECTS OF WORK BEHAVIOR , 1967 .

[119]  Paul D. Nelson,et al.  CRITERION MEASURES FOR EXTREMELY ISOLATED GROUPS , 1966 .

[120]  R. Cooper.  Leader's Task Relevance and Subordinate Behaviour in Industrial Work Groups , 1966 .

[121]  W. Sorenson.  Test of mechanical principles as a suppressor variable for the prediction of effectiveness on a mechanical repair job. , 1966 .

[122]  Louis P. Willemin.  Criterion Aspects of Army Research on the Prediction of Officer Performance , 1965 .

[123]  C. Dicken,et al.  PREDICTIVE VALIDITY OF PSYCHOMETRIC EVALUATIONS OF SUPERVISORS. , 1965, The Journal of applied psychology.

[124]  C. McDermid,et al.  SOME CORRELATES OF CREATIVITY IN ENGINEERING PERSONNEL. , 1965, The Journal of applied psychology.

[125]  E. Glaser,et al.  Validation of a multiple-assessment procedure for managerial personnel. , 1964 .

[126]  P. Nelson.  Supervisor esteem and personnel evaluations. , 1964 .

[127]  G. Forehand.  Assessments of innovative behavior: Partial criteria for the assessment of executive performance. , 1963 .

[128]  J. A. Hicks,et al.  The identification of traits related to managerial success. , 1962 .

[129]  E. Prien,et al.  ASSESSMENTS OF HIGHER-LEVEL PERSONNEL: III. RATING CRITERIA: A COMPARATIVE ANALYSIS OF SUPERVISOR RATINGS AND INCUMBENT SELF-RATINGS OF JOB PERFORMANCE , 1962 .

[130]  C. H. Lawshe,et al.  Foreman Leadership as Perceived By Superiors and Subordinates1 , 1959 .

[131]  Thomas B. Sprecher,et al.  A study of engineers' criteria for creativity. , 1959 .

[132]  J. W. Parker,et al.  Rating scale content: III. Relationship between supervisory- and self-ratings. , 1959 .

[133]  D. N. Buckner,et al.  THE PREDICTABILITY OF RATINGS AS A FUNCTION OF INTER-RATER AGREEMENT , 1959 .

[134]  J. T. Rusmore.  A Note on the “Test of Practical Judgment” , 1958 .

[135]  W. Mcgehee.  Are We Using What We Know about Training?=learning Theory and Training , 1958 .

[136]  R. Stoltz.  Development of a criterion of research productivity. , 1958 .

[137]  Kenneth F. Thomson,et al.  Prediction of the “Human Relations” Effectiveness of Industrial Supervisors1 , 1957 .

[138]  A. Spector.  Human relations behavior on the job: The officer behavior description. , 1957 .

[139]  A. Maslow,et al.  College teaching ability, scholarly activity and personality. , 1956 .

[140]  R. Stogdill,et al.  Leadership and role expectations , 1956 .

[141]  Carroll L. Shartle,et al.  Methods in the Study of Administrative Leadership , 1955 .

[142]  H. Strupp,et al.  Non‐Technical Factors in Supervisors' Ratings of Job Performance , 1955 .

[143]  Andrew C. Hilton,et al.  The validity of personnel assessments by professional psychologists. , 1955 .

[144]  E. Hay.  Comparative validities in clerical testing. , 1954 .

[145]  W. A. Klieger,et al.  The Effect of Opportunity to Observe and Rater Status on the Reliability of Performance Ratings1 , 1953 .

[146]  B. Bass,et al.  Effects of company rank on LGD performance of oil refinery supervisors. , 1953 .

[147]  C. H. Lawshe,et al.  Job performance criteria studies: I. The job performance of proofreaders. , 1951 .

[148]  J. Rupe.  When Workers Rate the Boss , 1951 .

[149]  Jurgensen Ce.  Overall job success as a basis for employee ratings. , 1950 .

[150]  E. B. Knauft.  A selection battery for bake shop managers. , 1949 .