A Meta-Analysis of Incremental Validity and Nomological Networks for Subordinate and Peer Rating

Use of subordinates and peers for job performance ratings has increased markedly in recent years, on the assumption that these sources cover different parts of the criterion space and provide incremental validity over and above supervisors. This study directly tested this assumption using a meta-analytically derived correlation matrix including supervisor-, subordinate-, and peer-ratings as well as objective performance measures. Hierarchical regression results showed that both subordinates and peers accounted for significant variance in objective measures, over and above other sources. Further meta-analyses investigated nomological networks of subordinate and peer ratings, specifically relations with personality and ability measures. Ability correlations for peers and especially subordinates were lower than those previously found for supervisors. Correlations for personality dimensions affiliation and agreeableness, although low, were somewhat higher than previous supervisor values. Results provide justification for the added expense of including subordinates and peers in a rating system.

[1]  D. Organ.  Organizational Citizenship Behavior: It's Construct Clean-Up Time , 1997 .

[2]  W. J. Mcnamara,et al.  A MANAGERIAL SELECTION STUDY , 1964 .

[3]  T. Judge,et al.  Personnel/Human Resources Management: A Political Influence Perspective , 1991 .

[4]  G. Neuman,et al.  Team effectiveness: beyond skills and cognitive ability. , 1999, The Journal of applied psychology.

[5]  C. McDermid,et al.  SOME CORRELATES OF CREATIVITY IN ENGINEERING PERSONNEL. , 1965, The Journal of applied psychology.

[6]  Walter C. Borman,et al.  360° ratings: An analysis of assumptions and a research agenda for evaluating their validity , 1997 .

[7]  D. P. Bozeman,et al.  Interrater agreement in multi‐source performance appraisal: a commentary , 1997 .

[8]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[9]  Steven E. Devlin,et al.  Predictive validity of biodata items generated from retrospective life experience essays. , 1990 .

[10]  W. McGaghie,et al.  Construct Validity of Medical Clinical Competence Measures: A Multitrait-Multimethod Matrix Study Using Confirmatory Factor Analysis , 1986 .

[11]  G. Mayo.  Peer Ratings and Halo , 1956 .

[12]  R. Wherry,et al.  Buddy Ratings: Popularity , 1949 .

[13]  L. Hough.  The 'Big Five' Personality Variables--Construct Confusion: Description Versus Prediction , 1992 .

[14]  J. Steyrer,et al.  Transformational Leadership and Objective Performance in Banks , 1998 .

[15]  Fred E. Fiedler,et al.  Leader intelligence and task performance: A test of a multiple screen model☆ , 1977 .

[16]  Leanne E. Atwater,et al.  Personal Attributes as Predictors of Superiors' and Subordinates' Perceptions of Military Academy Leadership , 1993 .

[17]  James W. Smither,et al.  CAN MULTI-SOURCE FEEDBACK CHANGE PERCEPTIONS OF GOAL ACCOMPLISHMENT, SELF-EVALUATIONS, AND PERFORMANCE-RELATED OUTCOMES? THEORY-BASED APPLICATIONS AND DIRECTIONS FOR RESEARCH , 1995 .

[18]  Kenneth F. Thomson,et al.  Prediction of the “Human Relations” Effectiveness of Industrial Supervisors1 , 1957 .

[19]  Leadership, organizational culture and organizational outcomes , 1992 .

[20]  Mina Westman,et al.  The Relationship Between Stress and Performance: The Moderating Effect of Hardiness , 1990 .

[21]  Kevin B. Lowe,et al.  Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the mlq literature , 1996 .

[22]  G. C. Carter.  Measurement of supervisory ability. , 1952 .

[23]  E. Lawler,et al.  Expectancy theory and job behavior , 1973 .

[24]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[25]  Carroll L. Shartle,et al.  Methods in the Study of Administrative Leadership , 1955 .

[26]  E. Morrison,et al.  Taking Charge At Work: Extrarole Efforts to Initiate Workplace Change , 1999 .

[27]  Allen I. Huffcutt,et al.  Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. , 1997 .

[28]  W. E. Scott,et al.  PSYCHOLOGICAL ATTRIBUTES OF EFFECTIVE LEADERSHIP IN A FORMAL ORGANIZATION , 1968 .

[29]  R. Wherry,et al.  Buddy Ratings: Popularity Contest or Leadership Criteria? , 1949 .

[30]  W. Borg.  Teacher Effectiveness in Team Teaching , 1967 .

[31]  John P. Campbell,et al.  A Confirmatory Test of a Model of Performance Determinants , 1994 .

[32]  John F. Kremer,et al.  Construct Validity of Multiple Measures in Teaching, Research, and Service and Reliability of Peer Ratings. , 1990 .

[33]  J. A. Hicks,et al.  The identification of traits related to managerial success. , 1962 .

[34]  W. Borman,et al.  Expanding the Criterion Domain to Include Elements of Contextual Performance , 1993 .

[35]  B. Bass,et al.  Transformational Leadership: Beyond Initiation and Consideration , 1990 .

[36]  H. Titus.  Prediction of supervisory success by use of standard psychological tests. , 1969 .

[37]  David M. Noer,et al.  Center for Creative Leadership , 1995 .

[38]  G. Curphy.  An empirical evaluation of Bass' (1985) theory of transformational and transactional leadership. , 1991 .

[39]  Frederick L. Oswald,et al.  Toward a Taxonomy of Managerial Performance Profiles , 1997 .

[40]  Murray R. Barrick,et al.  Validity of observer ratings of the big five personality factors , 1994 .

[41]  Philip L. Roth,et al.  DERIVATION AND IMPLICATIONS OF A META‐ANALYTIC MATRIX INCORPORATING COGNITIVE ABILITY, ALTERNATIVE PREDICTORS, AND JOB PERFORMANCE , 1999 .

[42]  Paul E. Levy,et al.  The Influence of Individual and Contextual Variables on 360-Degree Feedback System Attitudes , 1997 .

[43]  E. A. Fleishman,et al.  The description of supervisory behavior. , 1953 .

[44]  W. E. Holland,et al.  Individual characteristics of innovativeness and communication in research and development organizations. , 1978 .

[45]  Walter C. Borman,et al.  Effects of ratee task performance and interpersonal factors on supervisor and peer performance ratings , 1995 .

[46]  J. Conway,et al.  Distinguishing contextual performance from task performance for managerial jobs. , 1999 .

[47]  Bernard M. Bass,et al.  Long Term Forecasting of Transformational Leadership and Its Effects Among Naval Officers: Some Preliminary Findings , 1990 .

[48]  Dov Eden,et al.  Restraining the Golem: Boosting performance by changing the interpretation of low scores. , 1994 .

[49]  John Schaubroeck,et al.  A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. , 1988 .

[50]  James R. Van Scotter,et al.  Interpersonal facilitation and job dedication as separate facets of contextual performance. , 1996 .

[51]  L. Offermann,et al.  Transformational Leaders: Measurement of Personality Attributes and Work Group Performance , 1997 .

[52]  Robert Hogan,et al.  A Socioanalytic Perspective on Job Performance , 1998 .

[53]  Francis J. Yammarino,et al.  Do managers see themselves as other see them? Implications of self-other rating agreement for human resources management , 1997 .

[54]  Sarah A. Hezlett,et al.  TRAIT, RATER AND LEVEL EFFECTS IN 360‐DEGREE PERFORMANCE RATINGS , 1998 .

[55]  P. Mann.  Student consultants: Evaluations by consultees , 1973, American Journal of Community Psychology.

[56]  L. K. Waters,et al.  Peer nominations as predictors of short-term sales performance. , 1970, The Journal of applied psychology.

[57]  Paul E. Spector,et al.  Validity Generalization for Cognitive, Psychomotor, and Perceptual Tests for Craft Jobs in the Utility Industry , 1996 .

[58]  Walter W. Tornow,et al.  Perceptions or reality: Is multi-perspective measurement a means or an end? , 1993 .

[59]  Robert L. Heneman.  The relationship between supervisory ratings and results-oriented measures of performance: A meta-analysis. , 1986 .

[60]  D. Ones,et al.  Theory testing: Combining psychometric meta-analysis and structural equations modeling , 1995 .

[61]  Steven E. Scullen,et al.  Understanding the latent structure of job performance ratings. , 2000, The Journal of applied psychology.

[62]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[63]  Leslie Kanuk,et al.  LEADERSHIP BEHAVIOR AND EFFECTIVENESS OF BEAUTY SALON MANAGERS , 1979 .

[64]  Stephen A. Stumpf.  Career roles, psychological success, and job attitudes. , 1981 .

[65]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[66]  R. Sitgreaves.  Psychometric theory (2nd ed.). , 1979 .

[67]  Stanley B. Williams,et al.  Group opinion as a predictor of military leadership. , 1947, Journal of consulting psychology.

[68]  S. Fox,et al.  The Predictive Validity of the Sixteen Personality Factors Questionnaire Relative to Three Independent Criterion Measures of Military Performance , 1981 .

[69]  David W. Bracken,et al.  Multisource feedback: A study of its use in decision making , 1997 .

[70]  Francis J. Yammarino,et al.  DOES SELF‐OTHER AGREEMENT ON LEADERSHIP PERCEPTIONS MODERATE THE VALIDITY OF LEADERSHIP AND PERFORMANCE PREDICTIONS? , 1992 .

[71]  Walter C. Borman,et al.  The rating of individuals in organizations: An alternate approach , 1974 .

[72]  Eugene C. Mayfield.  Value of peer nominations in predicting life insurance sales performance. , 1972 .

[73]  W. F. Straub,et al.  Achieving consistency: A Study of basketball officiating , 1973 .

[74]  A. H. Church,et al.  Advancing the State of the Art of 360-Degree Feedback , 1997 .

[75]  Gregory J. Feist.  A Structural Model of Scientific Eminence , 1993 .

[76]  John W. Fleenor,et al.  Feedback to Managers: A Review and Comparison of Multi-Rater Instruments for Management Development. Third Edition. , 1998 .

[77]  Bradley R. Agle,et al.  CHARISMATIC CHIEF EXECUTIVE OFFICERS: ARE THEY MORE EFFECTIVE? AN EMPIRICAL TEST OF CHARISMATIC LEADERSHIP THEORY. , 1994 .

[78]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[79]  C. Dicken.  Predicting the success of Peace Corps community development workers. , 1969 .

[80]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[81]  John E. Hunter,et al.  Individual differences in output variability as a function of job complexity. , 1990 .

[82]  Jeffrey S. Kane,et al.  Methods of peer assessment. , 1978 .

[83]  Sonia S. Cowen.  A study of relationships between perceived leadership behaviors of presidents at public four-year institutions of higher education in the U.S. and the changes in FTE enrollment, perceptions of effectiveness, subordinate satisfaction, and other factors of the presidency. , 1992 .

[84]  Lawrence R. Jauch,et al.  Evaluation of University Professors' Research Performance , 1975 .

[85]  T. C. Parker.  Relationships among measures of supervisory behavior, group behavior, and situational characteristics. , 1963 .

[86]  A. H. Church,et al.  The relationship between individual personality orientation and executive leadership behaviour , 1998 .

[87]  W. Mischel,et al.  PREDICTING THE SUCCESS OF PEACE CORPS VOLUNTEERS IN NIGERIA. , 1965, Journal of personality and social psychology.

[88]  Jose M. Cortina,et al.  THE “BIG FIVE” PERSONALITY FACTORS IN THE IPI AND MMPI: PREDICTORS OF POLICE PERFORMANCE , 1992 .

[89]  J. Conway,et al.  Managerial Performance Development Constructs and Personality Correlates , 2000 .

[90]  Paul E. Tesluk,et al.  Personality as a Moderator in the Relationship Between Fairness and Retaliation , 1999 .

[91]  Neal Schmitt,et al.  Adverse impact and predictive efficiency of various predictor combinations , 1997 .

[92]  D. K. Trites.  Adaptability measures as predictors of performance ratings. , 1960 .

[93]  William H. Bommer,et al.  ON THE INTERCHANGEABILITY OF OBJECTIVE AND SUBJECTIVE MEASURES OF EMPLOYEE PERFORMANCE: A META-ANALYSIS , 1995 .

[94]  Harrison G. Gough,et al.  Testing for leadership with the California Psychological Inventory. , 1990 .

[95]  E. Hollander.  Peer nominations on leadership as a predictor of the pass-fail criterion in naval air training. , 1954 .

[96]  Laurence Siegel,et al.  PAIRED COMPARISON EVALUATIONS OF MANAGERIAL EFFECTIVENESS BY PEERS AND SUPERVISORS , 1982 .

[97]  C. I. Mosier,et al.  On the reliability of a weighted composite , 1943 .