Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment.

A quasi-experiment was conducted to investigate the effects of frame-of-reference training on the quality of competency modeling ratings made by consultants. Human resources consultants from a large consulting firm were randomly assigned to either a training or a control condition. The discriminant validity, interrater reliability, and accuracy of the competency ratings were significantly higher in the training group than in the control group. Further, the discriminant validity and interrater reliability of competency inferences were highest among an additional group of trained consultants who also had competency modeling experience. Together, these results suggest that procedural interventions such as rater training can significantly enhance the quality of competency modeling.

[1]  Robert W. Eichinger,et al.  The LEADERSHIP ARCHITECT norms and validity report , 2003 .

[2]  David J. Woehr,et al.  Understanding frame-of-reference training: the impact of training on the recall of performance information , 1994 .

[3]  R. Brennan.  Elements of generalizability theory , 1983 .

[4]  Juan I. Sanchez,et al.  The impact of raters' cognition on judgment accuracy: An extension to the job analysis domain , 1994 .

[5]  R. Lord.  Accuracy in behavioral measurement: An alternative definition based on raters' cognitive schema and signal detection theory. , 1985 .

[6]  H. John Bernardin,et al.  Strategies in Rater Training , 1981 .

[7]  Carol T. Kulik,et al.  Category‐Based and Feature‐Based Cognitive Processes in Job Impressions , 1993 .

[8]  David J. Woehr,et al.  Rater training for performance appraisal: A quantitative review , 1994 .

[9]  Ronald A. Ash,et al.  THE PRACTICE OF COMPETENCY MODELING , 2000 .

[10]  Mark S. Teachout,et al.  THE RELATIONSHIP BETWEEN WORK EXPERIENCE AND JOB PERFORMANCE: A CONCEPTUAL AND META‐ANALYTIC REVIEW , 1995 .

[11]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[12]  Michael A. Campion,et al.  Social and Cognitive Sources of Potential Inaccuracy in Job Analysis , 1997 .

[13]  Chet Robie,et al.  A new look at within-source interrater reliability of 360-degree feedback ratings. , 1998 .

[14]  Members of underrepresented groups: Reviewers for journal manuscripts wanted , 1989 .

[15]  Carol T. Kulik.  The Effects of Job Categorization on Judgments of the Motivating Potential of Jobs. , 1989 .

[16]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[17]  F. Morgeson,et al.  Self-presentation processes in job analysis: a field experiment investigating inflation in abilities, tasks, and competencies. , 2004, The Journal of applied psychology.

[18]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[19]  Michael A. Campion,et al.  Accuracy in job analysis: toward an inference‐based model , 2000 .

[20]  Milton D. Hakel,et al.  CONVERGENCE AMONG DATA SOURCES, RESPONSE BIAS, AND RELIABILITY AND VALIDITY OF A STRUCTURED JOB ANALYSIS QUESTIONNAIRE , 1979 .

[21]  Robert D. Gatewood,et al.  Human Resource Selection , 1997 .

[22]  Erich C. Dierdorff,et al.  A meta-analysis of job analysis reliability. , 2003, The Journal of applied psychology.

[23]  Filip Lievens,et al.  Inferring Competencies 2 Easing the Inferential Leap in Competency Modeling : The Effects of Task-Related Information and Subject Matter Expertise , 2004 .

[24]  S. Kay.  On the Nature of Expertise. , 1992 .

[25]  Robert L. Dipboye,et al.  Effects of training and information on the accuracy and reliability of job evaluations. , 1988 .

[26]  David V. Day,et al.  Effects of frame-of-reference training and information configuration on memory organization and rating accuracy. , 1995 .

[27]  David V. Day,et al.  Effects of frame-of-reference training on rater accuracy under alternative time delays , 1994 .

[28]  A. Denisi,et al.  Further investigation of common knowledge effects on job analysis ratings. , 1987 .

[29]  E. Levine,et al.  The Analysis of Work in the 20th and 21st Centuries , 2001 .

[30]  Thomas J. Keefe,et al.  Observational Purpose and Evaluative Articulation in Frame-of-Reference Training: The Effects of Alternative Processing Modes on Rating Accuracy , 1994 .

[31]  J. Salgado,et al.  Test–Retest Reliability of Ratings of Job Performance Dimensions in Managers , 2003 .

[32]  Juan I. Sanchez.  From documentation to innovation: Reshaping job analysis to meet emerging business needs , 1994 .

[33]  K. Murphy.  Personnel Selection in Organizations , 1992 .