WORK SAMPLE TESTS IN PERSONNEL SELECTION: A META-ANALYSIS OF BLACK–WHITE DIFFERENCES IN OVERALL AND EXERCISE SCORES

Work sample exams are generally thought to have either low or comparatively low levels of ethnic group differences when used for personnel selection. Such exams are sometimes called “simulation exercises” and involve having applicants perform a set of tasks that are similar to those performed on the job. The nearly ubiquitous meta-analytic value of Black–White subgroup differences in the literature is d= .38. Unfortunately, this estimate is plagued by a variety of problems (e.g., range restriction, inclusion of nonwork sample tests). Further, there are virtually no analyses that examine how the saturation of different constructs influence work sample tests. We gathered available data for Black–White ethnic group differences and found that overall work sample differences were markedly larger for samples of job applicants (d= .73) than previously thought. We also examined how different exercises and saturation of different constructs influenced work sample ds. For example, work sample test ratings of cognitive and job knowledge skills were associated with a mean observed d= .80, whereas ratings of various social skills were associated with mean observed ds that varied from .21 to .27. We urge scientists and practitioners to consider both the method and the constructs that are targeted when forecasting predictor ds.

[1]  Robert E. Ployhart,et al.  Determinants, Detection and Amelioration of Adverse Impact in Personnel Selection Procedures: Issues, Evidence and Lessons Learned , 2001 .

[2]  Ivan T. Robertson,et al.  Work sample tests: Validity, adverse impact and applicant reaction , 1982 .

[3]  John P. Campbell,et al.  A Confirmatory Test of a Model of Performance Determinants , 1994 .

[4]  L. Atwater,et al.  Does whom you work with matter? Effects of referent group gender and age composition on managers' compensation. , 2003, The Journal of applied psychology.

[5]  Luis Aramburu-Zabala Higuera,et al.  Adverse Impact in Personnel Selection: The Legal Framework and Test Bias , 2001 .

[6]  Allen I. Huffcutt,et al.  Ethnic group differences in measures of job performance: a new meta-analysis. , 2003, The Journal of applied psychology.

[7]  D. Chan.  Interactive effects of situational judgment effectiveness and proactive personality on work perceptions and work outcomes. , 2006, The Journal of applied psychology.

[8]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[9]  M. D. Dunnette,et al.  An alternative selection procedure: The low-fidelity simulation. , 1990 .

[10]  Huy Le,et al.  Increasing the Accuracy of Corrections for Range Restriction: Implications for Selection Procedure Validities and Other Research Results , 2006 .

[11]  P. Bobko,et al.  Work Sample Selection Tests and Expected Reduction in Adverse Impact: A Cautionary Note , 2005 .

[12]  Richard D. Arvey,et al.  MOTIVATIONAL COMPONENTS OF TEST TAKING , 1990 .

[13]  Filip Lievens,et al.  Predicting adverse impact and mean criterion performance in multistage selection. , 2006, The Journal of applied psychology.

[14]  S. J. Goff,et al.  Joint relation of experience and ability with job performance: Test of three hypotheses. , 1988 .

[15]  Allen I. Huffcutt,et al.  Identification and meta-analytic assessment of psychological constructs measured in employment interviews. , 2001, The Journal of applied psychology.

[16]  Fritz Drasgow,et al.  Bootstrap estimates of standard errors in validity generalization , 1992 .

[17]  John E. Hunter,et al.  Job sample vs. paper-and-pencil trades and technical tests: Adverse impact and examinee attitudes. , 1977 .

[18]  Douglas W. Bray,et al.  Validation of employment tests for telephone company installation and repair occupations. , 1970 .

[19]  Paul R. Sackett,et al.  Further exploration of typical and maximum performance criteria: definitional issues, prediction, and White−Black differences , 1993 .

[20]  Jill E. Ellingson,et al.  High-stakes testing in employment, credentialing, and higher education. Prospects in a post-affirmative-action world. , 2001, The American psychologist.

[21]  Paul R. Sackett,et al.  THE EFFECTS OF FORMING MULTI‐PREDICTOR COMPOSITES ON GROUP DIFFERENCES AND ADVERSE IMPACT , 1997 .

[22]  R. Ritchie,et al.  Employment test validation for minority and nonminority telephone operators. , 1975 .

[23]  Philip L. Roth,et al.  A META‐ANALYSIS OF WORK SAMPLE TEST VALIDITY: UPDATING AND INTEGRATING SOME CLASSIC LITERATURE , 2005 .

[24]  F. Thumin.  Predictor validity as related to criterion relevance, restriction of range, and ethnicity , 1993 .

[25]  Ability Test Equity in Predicting Job Performance Work Samples , 1996 .

[26]  N. Schmitt,et al.  An Evaluation of Two Strategies for Reducing Adverse Impact and Their Effects on Criterion-Related Validity , 1996 .

[27]  D. Lubinski.  Applied individual differences research and its quantitative methods. , 1996 .

[28]  Robert E. Ployhart,et al.  THE DIVERSITY–VALIDITY DILEMMA: STRATEGIES FOR REDUCING RACIOETHNIC AND SEX SUBGROUP DIFFERENCES AND ADVERSE IMPACT IN SELECTION , 2008 .

[29]  J. Sweller,et al.  Cognitive Load Theory and Complex Learning: Recent Developments and Future Directions , 2005 .

[30]  Robert E. Ployhart,et al.  Staffing in the 21st Century: New Challenges and Strategic Opportunities , 2006 .

[31]  J. G. Veres,et al.  Validation of a clerical test using work samples , 1992 .

[32]  Michael A. McDaniel,et al.  A reexamination of black-white mean differences in work performance: more data, more moderators. , 2006, The Journal of applied psychology.

[33]  P. Bobko,et al.  ETHNIC GROUP DIFFERENCES IN COGNITIVE ABILITY IN EMPLOYMENT AND EDUCATIONAL SETTINGS: A META‐ANALYSIS , 2001 .

[34]  Ivan T. Robertson,et al.  Work Sample Testing , 2000 .

[35]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[36]  Alice N. Outerbridge,et al.  Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance , 1986 .

[37]  Huy Le,et al.  Implications of direct and indirect range restriction for meta-analysis methods and findings. , 2006, The Journal of applied psychology.

[38]  Teva J. Scheer.  Uniform Guidelines on Employee Selection Procedures , 2007 .

[39]  Kenneth P. Yusko,et al.  THE ROLE OF COGNITIVE ABILITY IN THE SUBGROUP DIFFERENCES AND INCREMENTAL VALIDITY OF ASSESSMENT CENTER EXERCISES , 1998 .

[40]  H. J. Bernardin.  An Analysis of Black–White Differences in Job Performance. , 1984 .

[41]  R. Ritchie,et al.  Employment test validation for minority and nonminority clerks with work sample criteria. , 1975 .

[42]  Michael A. McDaniel,et al.  Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis , 2008 .

[43]  Walter C. Borman,et al.  Models of supervisory job performance ratings , 1991 .

[44]  Kenneth P. Yusko,et al.  EXPLORING BLACK‐WHITE SUBGROUP DIFFERENCES OF MANAGERIAL COMPETENCIES , 2001 .