Reviewing, Categorizing, and Analyzing the Literature on Black–White Mean Differences for Predictors of Job Performance: Verifying Some perceptions and Updating/Correcting Others

In both theoretical and applied literatures, there is confusion regarding accurate values for expected Black–White subgroup differences in personnel selection test scores. Much confusion arises because empirical estimates of standardized subgroup differences (d) are subject to many of the same biasing factors associated with validity coefficients (i.e., d is functionally related to a point-biserial r). To address such issues, we review/cumulate, categorize, and analyze a systematic set of many predictor-specific meta-analyses in the literature. We focus on confounds due to general use of concurrent, versus applicant, samples in the literature on Black–White d. We also focus on potential confusion due to different constructs being assessed within the same selection test method, as well as the influence of those constructs on d. It is shown that many types of predictors (such as biodata inventories or assessment centers) can have magnitudes of d that are much larger than previously thought. Indeed, some predictors (such as work samples) can have ds similar to that associated with paper-and-pencil tests of cognitive ability. We present more realistic values of d for both researcher and practitioner use. Implications for practice and future research are noted.

[1]  S. J. Motowidlo,et al.  Evidence that task performance should be distinguished from contextual performance. , 1994 .

[2]  Jeffrey M. Cucina,et al.  Forced-Choice Personality Tests: A Measure of Personality and Cognitive Ability? , 2006 .

[3]  Traditional tests and job simulations: minority and majority performance and test validities. , 2001 .

[4]  M. Mount,et al.  Validity of observer ratings of the five-factor model of personality traits: a meta-analysis. , 2011, The Journal of applied psychology.

[5]  John P. Hausknecht,et al.  Applicant Reactions to Selection Procedures: An Updated Model and Meta-Analysis , 2004 .

[6]  Patrick H. Raymark,et al.  The criterion-related validity of integrity tests: an updated meta-analysis. , 2012, The Journal of applied psychology.

[7]  Fred S. Switzer,et al.  PRIOR SELECTION CAUSES BIASED ESTIMATES OF STANDARDIZED ETHNIC GROUP DIFFERENCES: SIMULATION AND ANALYSIS , 2001 .

[8]  P. Bobko,et al.  College grade point average as a personnel selection device: ethnic group differences and potential adverse impact. , 2000, The Journal of applied psychology.

[9]  Gregory M. Hurtz,et al.  Personality and job performance: the Big Five revisited. , 2000, The Journal of applied psychology.

[10]  Jill C. Bradley,et al.  SITUATIONAL JUDGMENT TESTS: CONSTRUCTS ASSESSED AND A META‐ANALYSIS OF THEIR CRITERION‐RELATED VALIDITIES , 2010 .

[11]  G. C. Thornton,et al.  EXAMINING SELECTION UTILITY WHERE COMPETING PREDICTORS DIFFER IN ADVERSE IMPACT , 1997 .

[12]  Eric D. Heggestad,et al.  A Silk Purse From the Sow's Ear: Retrieving Normative Information From Multidimensional Forced-Choice Items , 2005 .

[13]  Thomas Bliesener Methodological moderators in validating biographical data in personnel selection1 , 1996 .

[14]  James Outtz The Role of Cognitive Ability Tests in Employment Selection , 2002 .

[15]  Robert E. Ployhart,et al.  THE DIVERSITY–VALIDITY DILEMMA: STRATEGIES FOR REDUCING RACIOETHNIC AND SEX SUBGROUP DIFFERENCES AND ADVERSE IMPACT IN SELECTION , 2008 .

[16]  Keith Hattrup,et al.  The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predicted performance. , 1997 .

[17]  Winfred Arthur,et al.  Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. , 1994 .

[18]  R. Landers,et al.  REVISITING INTERVIEW–COGNITIVE ABILITY RELATIONSHIPS: ATTENDING TO SPECIFIC RANGE RESTRICTION MECHANISMS IN META‐ANALYSIS , 2007 .

[19]  Michael D. Mumford,et al.  Assessing the Construct Validity of Rational Biodata Scales , 1995 .

[20]  John E. Hunter,et al.  Statistical power in criterion-related validation studies. , 1976 .

[21]  N. Schmitt,et al.  Developing a biodata measure and situational judgment inventory as predictors of college student performance. , 2004, The Journal of applied psychology.

[22]  Anton J. Villado,et al.  The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. , 2008, The Journal of applied psychology.

[23]  Michael A. McDaniel,et al.  Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis , 2008 .

[24]  P. Bobko,et al.  Forming Composites of Cognitive Ability and Alternative Measures to Predict Job Performance and Reduce Adverse Impact: Corrected Estimates and Realistic Expectations , 2005 .

[25]  Frederick L. Oswald,et al.  Personality Testing and Industrial–Organizational Psychology: Reflections, Progress, and Prospects , 2008, Industrial and Organizational Psychology.

[26]  N. Schmitt,et al.  Models of Job Performance Ratings: An Examination of Ratee Race, Ratee Gender, and Rater Level Effects , 1996 .

[27]  P. Sackett,et al.  Correction for range restriction: an expanded typology. , 2000, The Journal of applied psychology.

[28]  Amy L. Kristof PERSON-ORGANIZATION FIT: AN INTEGRATIVE REVIEW OF ITS CONCEPTUALIZATIONS, MEASUREMENT, AND IMPLICATIONS , 1996 .

[29]  Paul M. Muchinsky,et al.  The Correction for Attenuation , 1996 .

[30]  Philip L. Roth,et al.  Correcting the Effect Size of d for Range Restriction and Unreliability , 2001 .

[31]  Allen I. Huffcutt,et al.  Corrections for range restriction in structured interview ethnic group differences: the values may be larger than researchers thought. , 2002, The Journal of applied psychology.

[32]  R. L. Dipboye,et al.  RECONSIDERING THE USE OF PERSONALITY TESTS IN PERSONNEL SELECTION CONTEXTS , 2007 .

[33]  H. G. Osburn,et al.  Moderating effects of decision-making/information-processing job dimensions on test validities. , 1983 .

[34]  P. Bobko Correlation and Regression: Applications for Industrial Organizational Psychology and Management , 2001 .

[35]  Kenneth P. Yusko,et al.  EXPLORING BLACK‐WHITE SUBGROUP DIFFERENCES OF MANAGERIAL COMPETENCIES , 2001 .

[36]  T. Judge,et al.  IN SUPPORT OF PERSONALITY ASSESSMENT IN ORGANIZATIONAL SETTINGS , 2007 .

[37]  Eric Anthony Day,et al.  A META‐ANALYSIS OF THE CRITERION‐RELATED VALIDITY OF ASSESSMENT CENTER DIMENSIONS , 2003 .

[38]  Robert E. Ployhart,et al.  Personality and Situational Judgment Tests Across Applicant and Incumbent Settings: An Examination of Validity, Measurement, and Subgroup Differences , 2004 .

[39]  Allen I. Huffcutt,et al.  Ethnic group differences in measures of job performance: a new meta-analysis. , 2003, The Journal of applied psychology.

[40]  P. Bobko,et al.  ETHNIC GROUP DIFFERENCES IN COGNITIVE ABILITY IN EMPLOYMENT AND EDUCATIONAL SETTINGS: A META‐ANALYSIS , 2001 .

[41]  Matthew J Borneman,et al.  High stakes testing in higher education and employment: appraising the evidence for validity and fairness. , 2008, The American psychologist.

[42]  Ivan T. Robertson,et al.  Work Sample Testing , 2000 .

[43]  Robert E. Ployhart,et al.  Staffing in the 21st Century: New Challenges and Strategic Opportunities , 2006 .

[44]  Patrick D. Lynch,et al.  Perceived organizational support and police performance: the moderating influence of socioemotional needs. , 1998 .

[45]  Michael A. Campion,et al.  Use of situational judgment tests to predict job performance: a clarification of the literature. , 2001, The Journal of applied psychology.

[46]  Philip L. Roth,et al.  Racial Group Differences in Employment Interview Evaluations , 1998 .

[47]  James M. LeBreton,et al.  A Conditional Reasoning Measure for Aggression , 2005 .

[48]  N. Schmitt,et al.  Video-based versus paper-and-pencil method of assessment in situational judgment tests: subgroup differences in test performance and face validity perceptions. , 1997, The Journal of applied psychology.

[49]  P. Bobko,et al.  Updating the trainability tests literature on Black-White subgroup differences and reconsidering criterion-related validity. , 2011, The Journal of applied psychology.

[50]  Paul R. Sackett,et al.  MULTI‐STAGE SELECTION STRATEGIES: A MONTE CARLO INVESTIGATION OF EFFECTS ON PERFORMANCE AND MINORITY HIRING , 1996 .

[51]  Gary N. Burns,et al.  Reconsidering Forced-Choice Item Formats for Applicant Personality Assessment , 2005 .

[52]  Mark C. Bowler,et al.  Assessment center construct-related validity: Stepping beyond the MTMM matrix , 2009 .

[53]  Dan J. Putka,et al.  Reconsidering vocational interests for personnel selection: the validity of an interest-based selection test in relation to job knowledge, job performance, and continuance intentions. , 2011, The Journal of applied psychology.

[54]  G. V. Barrett Practitioner’s View of Personality Testing and Industrial–Organizational Psychology: Practical and Legal Issues , 2008, Industrial and Organizational Psychology.

[55]  R. McDaniel,et al.  Diversity as a Management Strategy for Organizations , 1997 .

[56]  Philip L. Roth,et al.  DERIVATION AND IMPLICATIONS OF A META‐ANALYTIC MATRIX INCORPORATING COGNITIVE ABILITY, ALTERNATIVE PREDICTORS, AND JOB PERFORMANCE , 1999 .

[57]  Filip Lievens,et al.  The operational validity of a video-based situational judgment test for medical college admissions: illustrating the importance of matching predictor and criterion construct domains. , 2005, The Journal of applied psychology.

[58]  Michael Matthews,et al.  Using Biodata to Predict Turnover, Organizational Commitment, and Job Performance in Healthcare , 2009 .

[59]  Chockalingam Viswesvaran,et al.  Gender, age, and race differences on overt integrity tests: Results across four large-scale job applicant datasets. , 1998 .

[60]  Kenneth P. Yusko,et al.  THE ROLE OF COGNITIVE ABILITY IN THE SUBGROUP DIFFERENCES AND INCREMENTAL VALIDITY OF ASSESSMENT CENTER EXERCISES , 1998 .

[61]  W. D. Corte Weighing job performance predictors to both maximize the quality of the selected workforce and control the level of adverse impact , 1999 .

[62]  P. Bobko,et al.  Work Sample Selection Tests and Expected Reduction in Adverse Impact: A Cautionary Note , 2005 .

[63]  Lois E. Tetrick,et al.  Society for Industrial and Organizational Psychology , 2010 .

[64]  F. Schmidt,et al.  Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. , 1993 .

[65]  Allen I. Huffcutt,et al.  Identification and meta-analytic assessment of psychological constructs measured in employment interviews. , 2001, The Journal of applied psychology.

[66]  Robert E. Ployhart,et al.  Determinants, Detection and Amelioration of Adverse Impact in Personnel Selection Procedures: Issues, Evidence and Lessons Learned , 2001 .

[67]  Jeffrey M. Cucina,et al.  Do warnings of response verification moderate the relationship between personality and cognitive ability? , 2005, The Journal of applied psychology.

[68]  Philip L. Roth,et al.  The Impact of Job Complexity and Study Design on Situational and Behavior Description Interview Validity , 2004 .

[69]  Leaetta M. Hough,et al.  Development and evaluation of the «accomplishment record» method of selecting and promoting professionals , 1984 .

[70]  G. Stokes,et al.  COMPARABILITY OF INCUMBENT AND APPLICANT SAMPLES FOR THE DEVELOPMENT OF BIODATA KEYS: THE INFLUENCE OF SOCIAL DESIRABILITY , 1993 .

[71]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[72]  Nathan S. Hartman,et al.  SITUATIONAL JUDGMENT TESTS, RESPONSE INSTRUCTIONS, AND VALIDITY: A META‐ANALYSIS , 2007 .

[73]  A. Ryan Explaining the Black-White Test Score Gap: The Role of Test Perceptions , 2001 .

[74]  Steffanie L. Wilk,et al.  Within-group norming and other forms of score adjustment in preemployment testing. , 1994, The American psychologist.

[75]  P. Bobko,et al.  Ethnic and gender subgroup differences in assessment center ratings: a meta-analysis. , 2008, The Journal of applied psychology.

[76]  Randall P. Settoon,et al.  Investigator characteristics as moderators of personnel selection research: a meta-analysis , 1994 .

[77]  Paul R. Sackett,et al.  Gravitation to jobs commensurate with ability: Longitudinal and cross-sectional tests. , 1995 .

[78]  Neal Schmitt,et al.  Incremental validity of situational judgment tests , 2001 .

[79]  Filip Lievens,et al.  Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. , 2007, The Journal of applied psychology.

[80]  Edwin E. Ghiselli,et al.  THE VALIDITY OF APTITUDE TESTS IN PERSONNEL SELECTION , 1973 .

[81]  Aparna Joshi,et al.  The Role Of Context In Work Team Diversity Research: A Meta-Analytic Review , 2009 .

[82]  R. Guion,et al.  A note on concurrent and predictive validity designs: A critical reanalysis. , 1982 .

[83]  K. Murphy,et al.  Controversy and consensus regarding the use of cognitive ability testing in organizations. , 2003, The Journal of applied psychology.

[84]  Bryan D. Edwards,et al.  Multistage selection strategies: simulating the effects on adverse impact and expected performance for various predictor combinations. , 2009, The Journal of applied psychology.

[85]  Paul R. Sackett,et al.  THE EFFECTS OF FORMING MULTI‐PREDICTOR COMPOSITES ON GROUP DIFFERENCES AND ADVERSE IMPACT , 1997 .

[86]  Bryan D. Edwards,et al.  MULTIPLE‐CHOICE AND CONSTRUCTED RESPONSE TESTS OF ABILITY: RACE‐BASED SUBGROUP PERFORMANCE DIFFERENCES ON ALTERNATIVE PAPER‐AND‐PENCIL TEST FORMATS , 2002 .

[87]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[88]  Philip L. Roth,et al.  WORK SAMPLE TESTS IN PERSONNEL SELECTION: A META-ANALYSIS OF BLACK–WHITE DIFFERENCES IN OVERALL AND EXERCISE SCORES , 2008 .

[89]  Jill E. Ellingson,et al.  High-stakes testing in employment, credentialing, and higher education. Prospects in a post-affirmative-action world. , 2001, The American psychologist.

[90]  Deniz S. Ones,et al.  GROUP DIFFERENCES IN PERSONALITY: META‐ANALYSES COMPARING FIVE U.S. RACIAL GROUPS , 2008 .

[91]  Deidra J. Schleicher,et al.  Can I retake it? Exploring subgroup differences and criterion-related validity in promotion retesting. , 2011, The Journal of applied psychology.

[92]  James S. Phillips,et al.  Concurrent and predictive validity designs: A critical reanalysis. , 1981 .

[93]  Bryan D. Edwards,et al.  An examination of factors contributing to a reduction in subgroup differences on a constructed-response paper-and-pencil test of scholastic achievement. , 2007, The Journal of applied psychology.

[94]  F. Oswald,et al.  Personnel selection: looking toward the future--remembering the past. , 2000, Annual review of psychology.