TRAIT, RATER AND LEVEL EFFECTS IN 360‐DEGREE PERFORMANCE RATINGS

Method and trait effects in multitrait-multirater (MTMR) data were examined in a sample of 2,350 managers who participated in a developmental feedback program. Managers rated their own performance and were also rated by two subordinates, two peers, and two bosses. The primary purpose of the study was to determine whether method effects are associated with the level of the rater (boss, peer, subordinate, self) or with each individual rater, or both. Previous research which has tacitly assumed that method effects are associated with the level of the rater has included only one rater from each level; consequently, method effects due to the rater's level may have been confounded with those due to the individual rater. Based on confirmatory factor analysis, the present results revealed that of the five models tested, the best fit was the 10-factor model which hypothesized 7 method factors (one for each individual rater) and 3 trait factors. These results suggest that method variance in MTMR data is more strongly associated with individual raters than with the rater's level. Implications for research and practice pertaining to multirater feedback programs are discussed.

[1]  Chockalingam Viswesvaran,et al.  Role of social desirability in personality testing for personnel selection: The red herring. , 1996 .

[2]  P. Bentler,et al.  Evaluating model fit. , 1995 .

[3]  Anne S. Tsui,et al.  MULTIPLE ASSESSMENT OF MANAGERIAL EFFECTIVENESS: INTERRATER AGREEMENT AND CONSENSUS IN EFFECTIVENESS MODELS , 1988 .

[4]  Joseph A. Cote,et al.  Additive and Multiplicative Method Effects in Applied Psychological Research: An Empirical Assessment of Three Models , 1994 .

[5]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[6]  N. Schmitt,et al.  Using the Lens Model to Magnify Raters' Consistency, Matching, and Shared Bias , 1986 .

[7]  P. Bentler,et al.  Comparative fit indexes in structural models. , 1990, Psychological bulletin.

[8]  Hannah R. Rothstein,et al.  Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. , 1990 .

[9]  K. Widaman.  Hierarchically Nested Covariance Structure Models for Multitrait-Multimethod Data , 1985 .

[10]  Richard W. Beatty,et al.  360‐degree feedback as a competitive advantage , 1993 .

[11]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[12]  Glenn M. McEvoy,et al.  Assessment centers and subordinate appraisals of managers: A seven-year examination of predictive validity. , 1989 .

[13]  H. Marsh.  Confirmatory Factor Analyses of Multitrait-Multimethod Data: Many Problems and a Few Solutions , 1989 .

[14]  Karl G. Jöreskog,et al.  Lisrel 8: User's Reference Guide , 1997 .

[15]  Walter W. Tornow,et al.  Perceptions or reality: Is multi-perspective measurement a means or an end? , 1993 .

[16]  Richard J. Klimoski,et al.  Role of the rater in performance appraisal. , 1974 .

[17]  Subordinates rating managers: Organizational and demographic correlates of self/subordinate agreement , 1993 .

[18]  Fritz Drasgow,et al.  A test of the measurement equivalence of the revised Job Diagnostic Survey: Past problems and current solutions. , 1988 .

[19]  Charles E. Lance,et al.  Statistical control of halo: Clarification from two cognitive models of the performance appraisal process. , 1986 .

[20]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[21]  N. Schmitt,et al.  RESEARCH NOTES. USING THE LENS MODEL TO MAGNIFY RATERS' CONSISTENCY, MATCHING, AND SHARED BIAS. , 1986 .

[22]  J. Conway,et al.  Analysis and Design of Multitrait-Multirater Performance Appraisal Studies , 1996 .

[23]  D. A. Kenny,et al.  Analysis of the multitrait^multimethod matrix by confirmatory factor analysis , 1992 .

[24]  Gina J. Medsker,et al.  A Review of Current Practices for Evaluating Causal Models in Organizational Behavior and Human Resources Management Research , 1994 .

[25]  Adrian Furnham,et al.  Congruence of self and subordinate ratings of managerial practices as a correlate of supervisor evaluation , 1994 .

[26]  Charles E. Lance,et al.  Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis. , 1992 .

[27]  R. P. McDonald,et al.  Goodness-of-fit indexes in confirmatory factor analysis : The effect of sample size , 1988 .

[28]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[29]  Kevin R. Murphy,et al.  Is halo error a property of the rater, ratees, or the specific behaviors observed? , 1992 .

[30]  M. Mount,et al.  Psychometric properties of subordinate ratings of managerial performance. , 1984 .

[31]  John Schaubroeck,et al.  A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. , 1988 .

[32]  Steven D. Maurer,et al.  The validity of employment interviews: A comprehensive review and meta-analysis. , 1994 .

[33]  Kurt Kraiger,et al.  A meta-analysis of ratee race effects in performance ratings. , 1985 .

[34]  P. Bentler,et al.  Significance Tests and Goodness of Fit in the Analysis of Covariance Structures , 1980 .

[35]  Sarah A. Hezlett,et al.  The impact of 360‐degree feedback on management skills development , 1993 .

[36]  R. Hoyle.  Structural equation modeling: concepts, issues, and applications , 1997 .

[37]  Francis J. Yammarino,et al.  DOES SELF‐OTHER AGREEMENT ON LEADERSHIP PERCEPTIONS MODERATE THE VALIDITY OF LEADERSHIP AND PERFORMANCE PREDICTIONS? , 1992 .

[38]  David D. Van Fleet,et al.  Theory and research on leadership in organizations. , 1992 .

[39]  E. Thorndike.  A constant error in psychological ratings. , 1920 .

[40]  James W. Smither,et al.  CAN MULTI-SOURCE FEEDBACK CHANGE PERCEPTIONS OF GOAL ACCOMPLISHMENT, SELF-EVALUATIONS, AND PERFORMANCE-RELATED OUTCOMES? THEORY-BASED APPLICATIONS AND DIRECTIONS FOR RESEARCH , 1995 .