Determinants of Interrater Reliability of Job Analysis: A Meta-analysis

Summary: This paper presents a meta-analysis of published job analysis interrater reliability data in order to predict the expected levels of interrater reliability within specific combinations of moderators, such as rater source, experience of the rater, and type of job descriptive information. The overall mean interrater reliability of 91 reliability coefficients reported in the literature was .59. The results of experienced professionals (job analysts) showed the highest reliability coefficients (.76). The method of data collection (job contact versus job description) only affected the results of experienced job analysts. For this group higher interrater reliability coefficients were obtained for analyses based on job contact (.87) than for those based on job descriptions (.71). For other rater categories (e.g., students, organization members) neither the method of data collection nor training had a significant effect on the interrater reliability. Analyses based on scales with defined levels resulted ...

[1]  L. R. Taylor.  EMPIRICALLY DERIVED JOB FAMILIES AS A FOUNDATION FOR THE STUDY OF VALIDITY GENERALIZATION , 1978 .

[2]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[3]  Edwin T. Cornelius,et al.  JOB ANALYSIS MODELS AND JOB CLASSIFICATION , 1979 .

[4]  Edward L. Levine,et al.  Determining important tasks within jobs: A policy-capturing approach. , 1989 .

[5]  Edwin T. Cornelius,et al.  A comparison of holistic and decomposed judgment strategies in job analyses by job incumbents. , 1980 .

[6]  Mark A. Wilson,et al.  The Validity of Task Coverage Ratings by Incumbents and Supervisors: Bad News , 1997 .

[7]  Robert J. Harvey,et al.  MONTE CARLO BASELINES FOR INTERRATER RELIABILITY CORRELATIONS USING THE POSITION ANALYSIS QUESTIONNAIRE , 1986 .

[8]  Robert J. Harvey,et al.  Influence of amount of job descriptive information on job analysis rating accuracy. , 1988 .

[9]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[10]  Ronald C. Page,et al.  A Comparison of the Practical Utility of Traditional, Statistical, and Hybrid Job Evaluation Approaches1 , 1982 .

[11]  E. Frieling,et al.  Some results with the German form of the Position Analysis Questionnaire (PAQ). , 1974 .

[12]  E. Lawler.  From job-based to competency-based organizations , 1994 .

[13]  Michael Smith.  A theory of the validity of predictors in selection , 1994 .

[14]  Robert C. Mecham,et al.  A study of job characteristics and job dimensions as based on the Position Analysis Questionnaire (PAQ). , 1972 .

[15]  Michael A. Surrette,et al.  Effects of analyst training and amount of available job related information on job analysis ratings , 1990 .

[16]  R. Rosenthal.  Meta-analytic procedures for social research , 1984 .

[17]  E. Prien,et al.  EVALUATION OF TASK AND JOB SKILL LINKAGE JUDGMENTS USED TO DEVELOP TEST SPECIFICATIONS , 1989 .

[18]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[19]  K. Murphy.  Personnel Selection in Organizations , 1992 .

[20]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[21]  Ivan T. Robertson,et al.  Advances in selection and assessment. , 1989 .

[22]  B. Green,et al.  Reliabilities of selected ratings available from the Dictionary of Occupational Titles.. , 1983 .

[23]  Brian Mullen,et al.  Advanced BASIC meta-analysis , 1989 .

[24]  Robert J. Harvey,et al.  A comparison of holistic versus decomposed rating of Position Analysis Questionnaire work dimensions. , 1988 .

[25]  Edwin A. Fleishman,et al.  Identifying Ability Requirements by Decision Flow Diagrams , 1980 .

[26]  Michael A. Campion,et al.  Social and Cognitive Sources of Potential Inaccuracy in Job Analysis , 1997 .

[27]  Robert L. Dipboye,et al.  Effects of training and information on the accuracy and reliability of job evaluations. , 1988 .

[28]  Sidney Gael.  The Job analysis handbook for business, industry, and government , 1988 .

[29]  P. Sparrow,et al.  Competency Requirement Forecasting: Issues for International Selection and Assessment , 1993 .

[30]  R. Madigan.  Comparable worth judgments: A measurement properties analysis. , 1985 .

[31]  Gregory A. Davis,et al.  Potential Sources of Bias in Job Analytic Processes1 , 1982 .

[32]  Steven L. Neuberg,et al.  A Continuum of Impression Formation, from Category-Based to Individuating Processes: Influences of Information and Motivation on Attention and Interpretation , 1990 .

[33]  Milton D. Hakel,et al.  CONVERGENCE AMONG DATA SOURCES, RESPONSE BIAS, AND RELIABILITY AND VALIDITY OF A STRUCTURED JOB ANALYSIS QUESTIONNAIRE , 1979 .

[34]  J. Sparrow.  The utility of PAQ in relating job behaviours to traits , 1989 .

[35]  Robert J. Harvey,et al.  CAN RATERS WITH REDUCED JOB DESCRIPTIVE INFORMATION PROVIDE ACCURATE POSITION ANALYSIS QUESTIONNAIRE (PAQ) RATINGS , 1986 .

[36]  Vandra L. Huber,et al.  Comparison of supervisor-incumbent and female-male multidimensional job evaluation ratings. , 1991 .

[37]  L. R. Taylor.  The Construction of Job Families Based on the Component and Overall Dimensions of the PAQ. , 1978 .

[38]  W. Sauser,et al.  EFFECTS OF ALTERNATIVE WEIGHTING METHODS IN A POLICY‐ CAPTURING APPROACH TO JOB EVALUATION: A REVIEW AND EMPIRICAL INVESTIGATION , 1991 .

[39]  C. Cooper,et al.  International review of industrial and organizational psychology , 1986 .

[40]  Dennis Doverspike,et al.  An internal bias analysis of a job evaluation instrument. , 1984 .

[41]  Wayne F. Cascio,et al.  Whither industrial and organizational psychology in a changing world of work , 1995 .

[42]  Donald P. Schwab,et al.  Assessment of a consensus-based multiple information source job evaluation system. , 1986 .

[43]  A. Denisi,et al.  Further investigation of common knowledge effects on job analysis ratings. , 1987 .

[44]  J. Earles,et al.  Methodology for Identifying Abilities for Job Classification: An Application of Job Analysis , 1996 .

[45]  John E. Hunter,et al.  Development of a general solution to the problem of validity generalization. , 1977 .

[46]  F. Schmidt,et al.  JOB CLASSIFICATION APPROACHES AND THE IMPLEMENTATION OF VALIDITY GENERALIZATION RESULTS , 1984 .

[47]  M. Gordon,et al.  The “Science of the Sophomore” Revisited: from Conjecture to Empiricism , 1986 .

[48]  Kenneth N. Wexley,et al.  An examination of differences between managerial effectiveness and response patterns on a structured job analysis questionnaire. , 1978 .

[49]  Juan I. Sanchez,et al.  The impact of raters' cognition on judgment accuracy: An extension to the job analysis domain , 1994 .

[50]  Steven F. Cronshaw,et al.  Generalizability analysis of a point method job evaluation instrument: A field study. , 1984 .

[51]  Dennis Doverspike,et al.  Generalizability analysis of a point-method job evaluation instrument. , 1983 .

[52]  Blair T. Johnson,et al.  Comparison of three major meta-analytic approaches. , 1995 .

[53]  Paul D. Geyer,et al.  RELIABILITIES OF RATINGS AVAILABLE FROM THE DICTIONARY OF OCCUPATIONAL TITLES , 1989 .

[54]  John E. Hunter,et al.  Employment testing: Old theories and new research findings. , 1981 .

[55]  Juan I. Sanchez,et al.  On the choice of scales for task analysis , 1992 .

[56]  Manuel London,et al.  RATINGS OF MANAGERIAL CHARACTERISTICS: EVALUATION DIFFICULTY, CO‐WORKER AGREEMENT, AND SELF‐AWARENESS , 1989 .

[57]  Edwin A. Fleishman,et al.  Reliability and validity of methods for evaluating perceived physical effort. , 1980 .

[58]  Ralph L. Rosnow,et al.  Essentials of Behavioral Research: Methods and Data Analysis , 1984 .

[59]  O. F. Voskuijl,et al.  The Use of Student Data in Job Analysis Research , 1999 .

[60]  Jack M. Feldman,et al.  Beyond Attribution Theory: Cognitive Processes in Performance Appraisal , 1981 .

[61]  Allan P. Jones,et al.  Narrative Job Descriptions as Potential Sources of Job Analysis Ratings. , 1982 .

[62]  Wayne F. Cascio,et al.  Applied Psychology in Human Resource Management , 1998 .

[63]  The Occupational Information Network: An Updated Dictionary of Occupational Titles , 1996 .

[64]  Thomas M. Stutzman,et al.  AN EVALUATION OF METHODS TO SELECT RESPONDENTS TO STRUCTURED JOB‐ANALYSIS QUESTIONNAIRES , 1986 .

[65]  Heinz Schuler.  Some advantages and problems of job analysis. , 1989 .

[66]  Thomas E. Powell,et al.  Ability-Requirement Factors for General Job Elements , 1996 .

[67]  Angelo S. DeNisi,et al.  EXPERT AND NAIVE RATERS USING THE PAQ: DOES IT MATTER? , 1984 .

[68]  N. Webb,et al.  Generalizability of General Education Development Ratings of Jobs in the United States , 1981 .

[69]  Steven D. Maurer,et al.  The validity of employment interviews: A comprehensive review and meta-analysis. , 1994 .

[70]  Donald B. Rubin,et al.  Interpersonal expectancy effects: the first 345 studies , 1978, Behavioral and Brain Sciences.