Self-other agreement in job performance ratings: a meta-analytic test of a process model.

This meta-analysis explores agreement in self- and supervisory ratings of job performance (k = 128 independent samples). It suggests a 3-stage model of the rating process and reviews the empirical evidence for the relevance of each of these 3 stages to an understanding of agreement in ratings. The proposed 3-stage model serves as the guiding rationale for the examination of an extensive set of variables that moderate rater agreement. Results are reported for 2 indicators of rater agreement (correlational and mean-level agreement). Self-supervisor ratings yielded an overall correlation of .22 (rho = .34; k = 115; n = 37,752). Position characteristics and the use of nonjudgmental performance indicators were the main moderators. Leniency in self-ratings is indicated by higher mean levels of self-ratings compared with supervisory ratings. Within Western samples, performance self-ratings showed leniency (d = 0.32, Delta = .49; k = 89; n = 35,417) dependent on contextual features, scale format, and scale content.

[1]  George C. Thornton,et al.  Effects of Gender on Self- and Supervisory Ratings , 1986 .

[2]  David E. Smith,et al.  A FIELD STUDY OF PERFORMANCE APPRAISAL PURPOSE: RESEARCH‐ VERSUS ADMINISTRATIVE‐ BASED RATINGS , 1995 .

[3]  Self‐ratings and supervisor ratings of graduate employees' competences during early career , 1992 .

[4]  A. Greenwald,et al.  Psychological perspectives on the self , 1982 .

[5]  L. Baird Self and Superior Ratings of Performance: As Related to Self-Esteem and Satisfaction with Supervision , 1977, IEEE Engineering Management Review.

[6]  Steven E. Scullen,et al.  Understanding the latent structure of job performance ratings. , 2000, The Journal of applied psychology.

[7]  Marilyn Aitkenhead Impression‐management and consistency effects in the processing of feedback , 1984 .

[8]  Shalom Malka,et al.  Application of the multitrait-multirater approach to performance appraisal in social service organizations , 1990 .

[9]  Christopher P. Parker,et al.  Reducing the Effects of Performance Expectations on Behavioral Ratings. , 2000, Organizational behavior and human decision processes.

[10]  Winston Bennett,et al.  Assessing measurement equivalence across rating sources: a multitrait-multirater approach. , 2005, The Journal of applied psychology.

[11]  Leanne E. Atwater,et al.  360° feedback: Accuracy, reactions, and perceptions of usefulness. , 2001 .

[12]  E. V. Hooft,et al.  Construct Validity of Multi-Source Performance Ratings: An Examination of the Relationship of Self-, Supervisor-, and Peer-Ratings with Cognitive and Personality Measures , 2006 .

[13]  John R. Weitzel,et al.  A TEST OF THE SITUATIONAL LEADERSHIP THEORY , 1990 .

[14]  A. Bedeian,et al.  AN EMPIRICAL INVESTIGATION OF SELF‐APPRAISAL‐BASED PERFORMANCE EVALUATION , 1988 .

[15]  James W. Smither,et al.  DOES PERFORMANCE IMPROVE FOLLOWING MULTISOURCE FEEDBACK? A THEORETICAL MODEL, META‐ANALYSIS, AND REVIEW OF EMPIRICAL FINDINGS , 2005 .

[16]  Nestor K. Ovalle,et al.  SELF‐APPRAISAL BASED UPON SUPERVISORY FEEDBACK , 1984 .

[17]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[18]  J Noak,et al.  Performance rating. , 1999, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[19]  I. M. Jawahar,et al.  WHERE ALL THE CHILDREN ARE ABOVE AVERAGE: THE PERFORMANCE APPRAISAL PURPOSE EFFECT , 1997 .

[20]  M. Sturman,et al.  The impact of job complexity and performance measurement on the temporal consistency, stability, and test-retest reliability of employee job performance ratings. , 2005, The Journal of applied psychology.

[21]  Vigrinia E. Hamori-Ota Gender differences in leadership style: Predictors of level of agreement between leader self-ratings and supervisory ratings, peer ratings, and ratings by direct reports. , 2007 .

[22]  Hart Blanton,et al.  Deviance Regulation: A Theory of Action and Identity , 2003 .

[23]  Fiona Patterson,et al.  Self-assessment accuracy and assessment centre decisions , 2000 .

[24]  A. Furnham,et al.  Congruence in Job-Performance Ratings: A Study of 360° Feedback Examining Self, Manager, Peers, and Consultant Ratings , 1998 .

[25]  K. Murphy,et al.  MODESTY BIAS IN SELF‐RATINGS OF PERFORMANCE: A TEST OF THE CULTURAL RELATIVITY HYPOTHESIS , 2006 .

[26]  Kevin R. Murphy,et al.  Understanding Performance Appraisal: Social, Organizational, and Goal-Based Perspectives , 1995 .

[27]  SINCE WHEN IS NO NEWS GOOD NEWS? THE RELATIONSHIP BETWEEN PERFORMANCE AND RESPONSE RATES IN MULTIRATER FEEDBACK , 2000 .

[28]  J. Werbel,et al.  Effects of Purpose of the Appraisal and Expectation of Validation On Self-Appraisal Leniency , 1986 .

[29]  Ronald B. Morgan SELF-AND CO-WORKER PERCEPTIONS OF ETHICS AND THEIR RELATIONSHIPS TO LEADERSHIP AND SALARY , 1993 .

[30]  R. Insinga,et al.  Self-Ratings of Workplace Behaviour: Contrasting Russia and Poland with the United States , 2005 .

[31]  Hun-Joon Park,et al.  REWARD ALLOCATIONS IN THE UNITED STATES, JAPAN, AND KOREA: A COMPARISON OF INDIVIDUALISTIC AND COLLECTIVISTIC CULTURES , 1990 .

[32]  Thomas E. Becker,et al.  A FIELD STUDY OF THE RELATIONSHIP BETWEEN THE ORGANIZATIONAL FEEDBACK ENVIRONMENT AND PERFORMANCE , 1989 .

[33]  Philip E. Tetlock,et al.  Impression management versus intrapsychic explanations in social psychology - a useful dichotomy , 1985 .

[34]  A. Suliman,et al.  Self and supervisor ratings of performance: Evidence from an individualistic culture , 2003 .

[35]  Jaap Scheerens,et al.  Measuring Change in Work Behavior by Means of Multisource Feedback , 2006 .

[36]  Robert T. Keller,et al.  The measurement of performance among research and development professional employees: A longitudinal analysis , 1982, IEEE Transactions on Engineering Management.

[37]  N. Schmitt,et al.  RESEARCH NOTES. USING THE LENS MODEL TO MAGNIFY RATERS' CONSISTENCY, MATCHING, AND SHARED BIAS. , 1986 .

[38]  Richard R. Reilly,et al.  AN EXAMINATION OF THE EFFECTS OF AN UPWARD FEEDBACK PROGRAM OVER TIME , 1995 .

[39]  Barry Gerhart,et al.  Sources of variance in incumbent perceptions of job complexity. , 1988 .

[40]  L. Abramson,et al.  Is there a universal positivity bias in attributions? A meta-analytic review of individual, developmental, and cultural differences in the self-serving attributional bias. , 2004, Psychological bulletin.

[41]  E. Perry,et al.  Performance appraisal and feedback: A consideration of national culture and a review of contemporary research and future trends. , 2002 .

[42]  D. A. Kenny,et al.  A general model of consensus and accuracy in interpersonal perception. , 1991, Psychological review.

[43]  A Ekpo-Ufot,et al.  Self-perceived task-relevant abilities, rated job performance, and complaining behavior of junior employees in a government ministry. , 1979, The Journal of applied psychology.

[44]  Donald G. Gardner,et al.  Focus of attention at work: Construct definition and empirical validation , 1989 .

[45]  John W. Fleenor,et al.  Does 360-degree Feedback Work in Different Industries? A Between-Industry Comparison of the Reliability and Validity of Multi-source Performance Ratings. , 1998 .

[46]  B. I. Van der Heijden,et al.  The value of subjectivity: problems and prospects for 360-degree appraisal systems , 2004 .

[47]  Cheri Ostroff,et al.  UNDERSTANDING SELF-OTHER AGREEMENT: A LOOK AT RATER AND RATEE CHARACTERISTICS, CONTEXT, AND OUTCOMES , 2004 .

[48]  W. E. Williams,et al.  Relationship between measures of effort and job performance. , 1973 .

[49]  Jack M. Feldman,et al.  Beyond Attribution Theory: Cognitive Processes in Performance Appraisal , 1981 .

[50]  P. C. Smith,et al.  Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. , 1963 .

[51]  E. Lawler,et al.  The multitrait-multirater approach to measuring managerial job performance. , 1967, The Journal of applied psychology.

[52]  J. Strauss Multi‐source perspectives of self‐esteem, performance ratings, and source agreement , 2005 .

[53]  Robert L. Holzbach,et al.  Rater bias in performance ratings: Superior, self-, and peer ratings. , 1978 .

[54]  P. Warr,et al.  Factors influencing two types of congruence in multirater judgments , 1999 .

[55]  B. R. Schlenker Impression Management: The Self-Concept, Social Identity, and Interpersonal Relations , 1980 .

[56]  M. Mount,et al.  Psychometric properties of subordinate ratings of managerial performance. , 1984 .

[57]  Chockalingam Viswesvaran,et al.  The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: unconfounding construct-level convergence and rating difficulty. , 2002, The Journal of applied psychology.

[58]  D. A. Kenny,et al.  Reconceptualizing individual differences in self-enhancement bias: an interpersonal approach. , 2004, Psychological review.

[59]  F. Yammarino,et al.  Performance in Relation to Job Skill Importance: A Consideration of Rater Source , 1993 .

[60]  Ronald A. Ash,et al.  Self-assessment in personnel selection. , 1977 .

[61]  Francis J. Yammarino,et al.  SELF‐OTHER AGREEMENT: DOES IT REALLY MATTER? , 1998 .

[62]  S. Fox,et al.  Variables affecting leniency, halo and validity of self‐appraisal , 1994 .

[63]  W. Borman,et al.  Expanding the Criterion Domain to Include Elements of Contextual Performance , 1993 .

[64]  Ramon J. Aldag,et al.  Moderators of the relationships between self and superior evaluations of job performance , 1977 .

[65]  John Schaubroeck,et al.  A meta-analysis of self-supervisor, self-peer, and peer-supervisor ratings. , 1988 .

[66]  L. Shore,et al.  Self- and supervisory perspectives on age and work attitudes and performance. , 1992, The Journal of applied psychology.

[67]  E. Prien,et al.  ASSESSMENTS OF HIGHER-LEVEL PERSONNEL: III. RATING CRITERIA: A COMPARATIVE ANALYSIS OF SUPERVISOR RATINGS AND INCUMBENT SELF-RATINGS OF JOB PERFORMANCE , 1962 .

[68]  S. J. Motowidlo,et al.  Relationship between self-rated performance and pay satisfaction among sales representatives. , 1982 .

[69]  K. Murphy Personnel Selection in Organizations , 1992 .

[70]  Bjørn Z. Ekelund,et al.  A study of the discrepancy between self- and observer-ratings on managerial derailment characteristics of European managers , 2007 .

[71]  Todd J. Maurer,et al.  Predictors of attitudes toward a 360‐degree feedback system and involvement in post‐feedback management development activity , 2002 .

[72]  A. Bandura Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[73]  William T. Hoyt,et al.  Rater bias in psychological research: when is it a problem and what can we do about it? , 2000, Psychological methods.

[74]  Robert D. Pritchard,et al.  The influence of valence, instrumentality, and expectancy on effort and performance. , 1973 .

[75]  Stephen G. West,et al.  Validity of self-evaluation of ability: A review and meta-analysis , 1982 .

[76]  Robert J. Wherry,et al.  THE CONTROL OF BIAS IN RATINGS: A THEORY OF RATING , 1982 .

[77]  Alan J. Dubinsky,et al.  An approach for assessing individual versus group effects in performance evaluations , 1987 .

[78]  B. Avolio,et al.  A MULTIPLE LEVEL INVESTIGATION OF PERSONNEL RATINGS , 1990 .

[79]  Lyman W. Porter,et al.  Managerial attitudes and performance , 1968 .

[80]  L. Hedges Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[81]  John W. Fleenor,et al.  Feedback to Managers: A Review and Comparison of Multi-Rater Instruments for Management Development. Third Edition. , 1998 .

[82]  Chockalingam Viswesvaran,et al.  RELIABILITY IS NOT VALIDITY AND VALIDITY IS NOT RELIABILITY , 2000 .

[83]  S. Nealey,et al.  A multitrait-multimethod analysis of predictors and criteria of nursing performance☆ , 1970 .

[84]  Sarah A. Hezlett,et al.  TRAIT, RATER AND LEVEL EFFECTS IN 360‐DEGREE PERFORMANCE RATINGS , 1998 .

[85]  Hannah R. Rothstein,et al.  Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe. , 1990 .

[86]  H. G. Heneman,et al.  Comparisons of self- and superior ratings of managerial performance. , 1974 .

[87]  E. E. Jones,et al.  When Self-Presentation is Constrained by the Target's Knowledge: Consistency and Compensation. , 1978 .

[88]  Clive Fletcher,et al.  A study of individual differences and self‐awareness in the context of multi‐source feedback , 2000 .

[89]  Eric Sundstrom,et al.  Office Noise, Satisfaction, and Performance , 1994 .

[90]  K. Murphy,et al.  Psychological Testing: Principles and Applications , 1988 .

[91]  James M. Diefendorff,et al.  The Impact of Contextual Self‐Ratings and Observer Ratings of Personality on the Personality–Performance Relationship1 , 2006 .

[92]  Keith Leavitt,et al.  Reducing the performance-cue bias in work behavior ratings: can groups help? , 2002, The Journal of applied psychology.

[93]  R. Goffin,et al.  The self‐rater's personality and self‐other disagreement in multi‐source performance ratings: Is disagreement healthy? , 2007 .

[94]  P. Warr,et al.  Associations between rating content and self-other agreement in multi-source feedback , 2000 .

[95]  James W. Smither,et al.  CAN MULTI-SOURCE FEEDBACK CHANGE PERCEPTIONS OF GOAL ACCOMPLISHMENT, SELF-EVALUATIONS, AND PERFORMANCE-RELATED OUTCOMES? THEORY-BASED APPLICATIONS AND DIRECTIONS FOR RESEARCH , 1995 .

[96]  C. R. Snyder,et al.  Effects of source status and message favorability on acceptance of personality feedback. , 1976, The Journal of applied psychology.

[97]  J. M. Geringer,et al.  Self-management training for improving job performance: a field experiment involving salespeople. , 2000, The Journal of applied psychology.

[98]  Subordinates rating managers: Organizational and demographic correlates of self/subordinate agreement , 1993 .

[99]  P. Williams Practice management. , 2005, The Alpha omegan.

[100]  Adrian Furnham,et al.  Congruence of self and subordinate ratings of managerial practices as a correlate of supervisor evaluation , 1994 .

[101]  D. R. Lehman,et al.  Is there a universal need for positive self-regard? , 1999, Psychological review.

[102]  Nigel Harvey,et al.  Blackwell Handbook of Judgment and Decision Making , 2004 .

[103]  D. Cervone,et al.  Using a knowledge-and-appraisal model of personality architecture to understand consistency and variability in smokers' self-efficacy appraisals in high-risk situations. , 2007, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[104]  Edward E. Lawler,et al.  A correlational-causal analysis of the relationship between expectancy attitudes and job performance. , 1968 .

[105]  Benedikt Hell,et al.  Konzeption und Prüfung eines multimodalen Systems der Leistungsbeurteilung : Individualmodul , 2003 .

[106]  Peter Warr,et al.  Logical and judgmental moderators of the criterion‐related validity of personality scales , 1999 .

[107]  C. Fletcher,et al.  The Psychometric Properties of 360 Degree Feedback: An Empirical Study and a Cautionary Tale , 1998 .

[108]  B. Heijden Age and Assessments of Professional Expertise: The Relationship between Higher Level Employees'Age and Self-assessments or Supervisor Ratings of Professional Expertise , 2001 .

[109]  John McENERY,et al.  Self‐rating in management training needs assessment: A neglected opportunity? , 1987 .

[110]  Barry M. Staw,et al.  Attribution of the "causes" of performance: A general alternative interpretation of cross-sectional research on organizations. , 1975 .

[111]  L. V. Dyne,et al.  Helping and Voice Extra-Role Behaviors: Evidence of Construct and Predictive Validity , 1998 .

[112]  Dirk D. Steiner,et al.  Common comparison standards : An approach to improving agreement between self and supervisory performance ratings , 1996 .

[113]  Francis J. Yammarino,et al.  DOES SELF‐OTHER AGREEMENT ON LEADERSHIP PERCEPTIONS MODERATE THE VALIDITY OF LEADERSHIP AND PERFORMANCE PREDICTIONS? , 1992 .

[114]  N. Schmitt,et al.  Using the Lens Model to Magnify Raters' Consistency, Matching, and Shared Bias , 1986 .

[115]  L. Mann,et al.  Leadership, Managerial Performance and 360-Degree Feedback , 1998 .

[116]  S. Carroll,et al.  Individualism/Collectivism Orientations and Reactions Toward Alternative Human Resource Management Practices , 1998 .

[117]  P. Levy,et al.  THE EFFECTS OF PERCEIVED SYSTEM KNOWLEDGE ON THE AGREEMENT BETWEEN SELF-RATINGS AND SUPERVISOR RATINGS , 2006 .

[118]  William M. Goldstein,et al.  Social Judgment Theory: Applying and Extending Brunswik's Probabilistic Functionalism , 2008 .

[119]  J. Conway,et al.  Distinguishing contextual performance from task performance for managerial jobs. , 1999 .

[120]  Donald J. Campbell,et al.  Self-Appraisal in Performance Evaluation: Development Versus Evaluation , 1988 .

[121]  S. J. Motowidlo,et al.  Evidence that task performance should be distinguished from contextual performance. , 1994 .

[122]  Mark John Somers,et al.  Assessing Self-Appraisal of Job Performance as an Evaluation Device: Are the Poor Results a Function of Method or Methodology? , 1991 .

[123]  J. W. Parker,et al.  Rating scale content: III. Relationship between supervisory- and self-ratings. , 1959 .

[124]  Jane R. Williams,et al.  Self‐Supervisor Agreement: The Influence of Feedback Seeking on the Relationship Between Self and Supervisor Ratings of Performance1 , 2000 .

[125]  Jiing-Lih Farh,et al.  Effects of Self-Esteem on Leniency Bias in Self-Reports of Performance: A Structural Equation Model Analysis , 2007 .

[126]  K. Moser,et al.  Feedback als Moderator der Validität von Selbstbeurteilungen , 2005 .

[127]  Joseph G. Rosse,et al.  Reconsidering the vertical dyad linkage model of leadership. , 1983 .

[128]  M. Audrey Korsgaard,et al.  THE EFFECT OF OTHER ORIENTATION ON SELF-SUPERVISOR RATING AGREEMENT , 2004 .

[129]  James M. LeBreton,et al.  The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar? , 2003 .

[130]  G. R. Ferris,et al.  THE INFLUENCE OF SUBORDINATE AGE ON PERFORMANCE RATINGS AND CAUSAL ATTRIBUTIONS , 1985 .

[131]  Zeynep Aycan,et al.  Cross-cultural industrial and organizational psychology: A critical appraisal of the field and future directions. , 2002 .

[132]  V B CLINE,et al.  INTERPERSONAL PERCEPTION. , 1964, Progress in experimental personality research.

[133]  J. Diefendorff,et al.  Measurement Equivalence and Multisource Ratings for Non-managerial Positions: Recommendations for Research and Practice , 2005 .

[134]  Allen I. Huffcutt,et al.  Psychometric properties of multisource performance ratings: A meta-analysis of subordinate, supervisor, peer, and self-ratings. , 1997 .

[135]  M. Ross,et al.  From chump to champ: people's appraisals of their earlier and present selves. , 2001, Journal of personality and social psychology.

[136]  Bor-Shiuan Cheng,et al.  Cultural Relativity in Action: A Comparison of Self-Ratings Made by Chinese and U.S. Workers , 1991 .

[137]  F. Yammarino,et al.  Congruence of Self and Others' Leadership Ratings of Naval Officers for Understanding Successful Performance , 1991 .

[138]  A. Wheeler,et al.  Self, Teacher, and Faculty Assessments of Student Teaching Performance. , 1982 .

[139]  P. Herriot,et al.  Self‐ratings, supervisor ratings, positions and performance , 1990 .

[140]  Gordon W. Cheung Multifaceted conceptions of self–other ratings disagreement. , 1999 .

[141]  James W. Smither,et al.  Accountability: The Achilles' Heel of Multisource Feedback , 1997 .

[142]  Robert J. Vance,et al.  Construct Validity of Three Types of Organizational Citizenship behavior: An Illustration of the Direct Product Model with Refinements , 1993 .

[143]  Richard J. Klimoski,et al.  Role of the rater in performance appraisal. , 1974 .

[144]  Frank J. Landy,et al.  The measurement of work performance : methods, theory and applications / Frank J. Landy, James L. Farr , 1983 .

[145]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[146]  Manuel London,et al.  RATINGS OF MANAGERIAL CHARACTERISTICS: EVALUATION DIFFICULTY, CO‐WORKER AGREEMENT, AND SELF‐AWARENESS , 1989 .

[147]  R. Kahn,et al.  Organizational psychology. , 1967, Annual review of psychology.

[148]  T. DeCotiis,et al.  The performance appraisal process: a model and some testable propositions. , 1978, Academy of management review. Academy of Management.

[149]  A. H. Church Do You See What I See? An Exploration of Congruence in Ratings From Multiple Perspectives1 , 1997 .

[150]  M. Lindeman,et al.  Acquaintanceship and the discrepancy between supervisor and self-assessments. , 1998 .

[151]  C. E. Willis,et al.  A methodological and substantive note on the performance-cue effect in ratings of work-group behavior. , 1995 .

[152]  Margaret C. Lohman The development of a multirater instrument for assessing employee problem‐solving skill , 2004 .

[153]  Barry R. Nathan,et al.  A comparison of validation criteria: Objective versus subjective performance measures and self- versus supervisor ratings. , 2006 .

[154]  Cheri Ostroff TRAINING EFFECTIVENESS MEASURES AND SCORING SCHEMES: A COMPARISON , 2006 .

[155]  Mary Jo Clark,et al.  An Assessment of Faculty Performance: Some Correlates Between Administrator, Colleague, Student and Self-Ratings. , 1975 .

[156]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[157]  H. Cooper Synthesizing Research: A Guide for Literature Reviews , 1998 .

[158]  Kevin R. Murphy,et al.  INTERRATER CORRELATIONS DO NOT ESTIMATE THE RELIABILITY OF JOB PERFORMANCE RATINGS , 2000 .

[159]  Neil M. A. Hauenstein An information-processing approach to leniency in performance judgments , 1992 .

[160]  W. B. Webb,et al.  Student, supervisor, and self-ratings of instructional proficiency. , 1955 .

[161]  George C. Thornton,et al.  THE RELATIONSHIP BETWEEN SUPERVISORY- AND SELF-APPRAISALS OF EXECUTIVE PERFORMANCE , 1968 .

[162]  William T. Hoyt,et al.  Magnitude and moderators of bias in observer ratings: A meta-analysis. , 1999 .

[163]  Robert E. Wood,et al.  Self- versus others' ratings as predictors of assessment center ratings: Validation evidence for 360-degree feedback programs. , 2002 .

[164]  Terry A. Beehr,et al.  Evaluation of 360 degree feedback ratings: relationships with each other and with performance and selection predictors† , 2001 .

[165]  Angelo S. DeNisi,et al.  A cognitive view of the performance appraisal process: A model and research propositions , 1984 .

[166]  Bernhard Wilpert,et al.  Applied psychology : an international review , 1989 .

[167]  Shaiful Annuar Khalid,et al.  Self and Superior Ratings of Organizational Citizenship Behavior: Are there Differences in the Source of Ratings? , 2005 .

[168]  Shaul Fox,et al.  VALIDITY OF SELF‐ASSESSMENT: A FIELD EVALUATION , 1988 .