Toward Better Meta-Analytic Matrices: How Input Values Can Affect Research Conclusions in Human Resource Management Simulations

Simulations and analyses based on meta-analytic matrices are fairly common in human resource management and organizational behavior research, particularly in staffing research. Unfortunately, the meta-analytic values estimates for validity and group differences (i.e., ρ and δ, respectively) used in such matrices often vary in the extent to which they are affected by artifacts and how accurately the values capture the underlying constructs and the appropriate population. We investigate how such concerns might influence conclusions concerning key issues such as prediction of job performance and adverse impact of selection procedures, as well as noting wider applications of these issues. We also start the process of building a better matrix upon which to base many such simulations and analyses in staffing research. Finally, we offer guidelines to help researchers/practitioners better model human resources processes, and we suggest ways that researchers in a variety of areas can better assemble meta-analytic matrices.

[1]  In-Sue Oh,et al.  Fixed- versus random-effects models in meta-analysis: model properties and an empirical comparison of differences in results. , 2009, The British journal of mathematical and statistical psychology.

[2]  D. Ones,et al.  Theory testing: Combining psychometric meta-analysis and structural equations modeling , 1995 .

[3]  A. Ryan,et al.  Variance in faking across noncognitive measures. , 2000, The Journal of applied psychology.

[4]  M. Mount,et al.  Validity of observer ratings of the five-factor model of personality traits: a meta-analysis. , 2011, The Journal of applied psychology.

[5]  J. Boudreau,et al.  Decision-theoretic utility analysis applied to employee separations and acquisitions. , 1985 .

[6]  Michael A. McDaniel,et al.  A reexamination of black-white mean differences in work performance: more data, more moderators. , 2006, The Journal of applied psychology.

[7]  Jeff W. Johnson,et al.  The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance. , 2001, The Journal of applied psychology.

[8]  Allen I. Huffcutt,et al.  Corrections for range restriction in structured interview ethnic group differences: the values may be larger than researchers thought. , 2002, The Journal of applied psychology.

[9]  F. Lievens,et al.  A Practical Procedure to Estimate the Quality and the Adverse Impact of Single-Stage Selection Decisions , 2003 .

[10]  Filip Lievens,et al.  A cautionary note on the effects of range restriction on predictor intercorrelations. , 2007, The Journal of applied psychology.

[11]  Gregory M. Hurtz,et al.  Personality and job performance: the Big Five revisited. , 2000, The Journal of applied psychology.

[12]  Chockalingam Viswesvaran,et al.  Gender, age, and race differences on overt integrity tests , 1998 .

[13]  W. D. Corte Weighing job performance predictors to both maximize the quality of the selected workforce and control the level of adverse impact , 1999 .

[14]  John W. Boudreau,et al.  THE EFFECTS OF VARIABILITY AND RISK IN SELECTION UTILITY ANALYSIS: AN EMPIRICAL COMPARISON , 1987 .

[15]  P. Bobko,et al.  ETHNIC GROUP DIFFERENCES IN COGNITIVE ABILITY IN EMPLOYMENT AND EDUCATIONAL SETTINGS: A META‐ANALYSIS , 2001 .

[16]  Neal Schmitt,et al.  Adverse impact and predictive efficiency of various predictor combinations , 1997 .

[17]  Filip Lievens,et al.  Predicting adverse impact and mean criterion performance in multistage selection. , 2006, The Journal of applied psychology.

[18]  Kruuk Hunter and Hunted , 2002 .

[19]  Daniel J. Svyantek,et al.  Analyzing meta-analysis: Potential problems, an unsuccessful replication, and evaluation criteria. , 1985 .

[20]  R. Landers,et al.  REVISITING INTERVIEW–COGNITIVE ABILITY RELATIONSHIPS: ATTENDING TO SPECIFIC RANGE RESTRICTION MECHANISMS IN META‐ANALYSIS , 2007 .

[21]  Robert E. Ployhart,et al.  THE DIVERSITY–VALIDITY DILEMMA: STRATEGIES FOR REDUCING RACIOETHNIC AND SEX SUBGROUP DIFFERENCES AND ADVERSE IMPACT IN SELECTION , 2008 .

[22]  Dennis Doverspike,et al.  Simulations as a Method of Illustrating the Impact of Differential Weights on Personnel Selection Outcomes , 1996 .

[23]  Patrick H. Raymark,et al.  What Do Structured Selection Interviews Really Measure? The Construct Validity of Behavior Description Interviews , 2004 .

[24]  F. Schmidt,et al.  Comprehensive meta-analysis of integrity test validities: Findings and implications for personnel selection and theories of job performance. , 1993 .

[25]  Paul R. Sackett,et al.  MULTI‐STAGE SELECTION STRATEGIES: A MONTE CARLO INVESTIGATION OF EFFECTS ON PERFORMANCE AND MINORITY HIRING , 1996 .

[26]  Amy L. Kristof PERSON-ORGANIZATION FIT: AN INTEGRATIVE REVIEW OF ITS CONCEPTUALIZATIONS, MEASUREMENT, AND IMPLICATIONS , 1996 .

[27]  Silvia Moscoso,et al.  Comprehensive meta-analysis of the construct validity of the employment interview , 2002 .

[28]  Chockalingam Viswesvaran,et al.  Gender, age, and race differences on overt integrity tests: Results across four large-scale job applicant datasets. , 1998 .

[29]  Jill E. Ellingson,et al.  No easy solution to subgroup differences. , 2002, The American psychologist.

[30]  G. Bower,et al.  The evolution of a cognitive psychologist: a journey from simple behaviors to complex mental acts. , 2008, Annual review of psychology.

[31]  Philip L. Roth,et al.  DERIVATION AND IMPLICATIONS OF A META‐ANALYTIC MATRIX INCORPORATING COGNITIVE ABILITY, ALTERNATIVE PREDICTORS, AND JOB PERFORMANCE , 1999 .

[32]  P. Sackett,et al.  FAKING IN PERSONNEL SELECTION: TRADEOFFS IN PERFORMANCE VERSUS FAIRNESS RESULTING FROM TWO CUT‐SCORE STRATEGIES , 2009 .

[33]  D. A. Seaver,et al.  A comparison of weight approximation techniques in multiattribute utility decision making , 1981 .

[34]  Kevin R. Murphy,et al.  Implications of the multidimensional nature of job performance for the validity of selection tests , 1997 .

[35]  P. Bobko,et al.  Forming Composites of Cognitive Ability and Alternative Measures to Predict Job Performance and Reduce Adverse Impact: Corrected Estimates and Realistic Expectations , 2005 .

[36]  Murray R. Barrick,et al.  THE BIG FIVE PERSONALITY DIMENSIONS AND JOB PERFORMANCE: A META-ANALYSIS , 1991 .

[37]  Herman Aguinis,et al.  Meta-Analytic Choices and Judgment Calls: Implications for Theory Building and Testing, Obtained Effect Sizes, and Scholarly Impact , 2011 .

[38]  Chet Robie,et al.  Faking and the validity of conscientiousness: a Monte Carlo investigation. , 2008, The Journal of applied psychology.

[39]  N. Schmitt,et al.  The impact of corrections for faking on the validity of noncognitive measures in selection settings. , 2006, The Journal of applied psychology.

[40]  Neil Anderson,et al.  A meta-analytic study of general mental ability validity for different occupations in the European community. , 2003, The Journal of applied psychology.

[41]  Frank L. Schmidt,et al.  Increased Accuracy for Range Restriction Corrections: Implications for the Role of Personality and General Mental Ability in Job and Training Performance , 2008 .

[42]  Paul K. Bergey,et al.  FORCED DISTRIBUTION RATING SYSTEMS AND THE IMPROVEMENT OF WORKFORCE POTENTIAL: A BASELINE SIMULATION , 2005 .

[43]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[44]  Michael A. McDaniel,et al.  A meta-analytic investigation of cognitive ability in employment interview evaluations: Moderating characteristics and implications for incremental validity. , 1996 .

[45]  Stephanie C Payne,et al.  A meta-analytic examination of the goal orientation nomological net. , 2007, The Journal of applied psychology.

[46]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[47]  Philip L. Roth,et al.  WORK SAMPLE TESTS IN PERSONNEL SELECTION: A META-ANALYSIS OF BLACK–WHITE DIFFERENCES IN OVERALL AND EXERCISE SCORES , 2008 .

[48]  Nathan S. Hartman,et al.  SITUATIONAL JUDGMENT TESTS, RESPONSE INSTRUCTIONS, AND VALIDITY: A META‐ANALYSIS , 2007 .

[49]  F. Schmidt,et al.  Selection utility in the occupation of U.S. Park ranger for three modes of test use , 1984 .

[50]  S. Zedeck Fairness in Employment Testing: Validity Generalization, Minority Issues, and the General Aptitude Test Battery. , 1990 .

[51]  John A. Henderson,et al.  Is utility really futile? A failure to replicate and an extension. , 1998 .

[52]  F. Lievens,et al.  Personnel selection. , 2008, Annual review of psychology.

[53]  J. Hunter Cognitive ability, cognitive aptitudes, job knowledge, and job performance , 1986 .

[54]  Michael A. McDaniel,et al.  Use of situational judgment tests to predict job performance: a clarification of the literature. , 2001, The Journal of applied psychology.

[55]  D. Ones The construct validity of integrity tests. , 1994 .

[56]  Philip L. Roth,et al.  A META‐ANALYSIS OF WORK SAMPLE TEST VALIDITY: UPDATING AND INTEGRATING SOME CLASSIC LITERATURE , 2005 .

[57]  Herman Aguinis,et al.  Customer-Centric Science: Reporting Significant Research Results With Rigor, Relevance, and Practical Impact in Mind , 2010 .

[58]  Michael A. McDaniel,et al.  PUBLICATION BIAS: A CASE STUDY OF FOUR TEST VENDORS , 2006 .

[59]  Herman Aguinis,et al.  Understanding the impact of test validity and bias on selection errors and adverse impact in human resource selection. , 2007 .

[60]  Keith Hattrup,et al.  A Comparison of Predictor-Based and Criterion-Based Methods for Weighing Predictors to Reduce Adverse Impact , 2002 .

[61]  Timothy J. Pleskac,et al.  Prediction of 4-year college student performance using cognitive and noncognitive predictors and the impact on demographic status of admitted students. , 2009, The Journal of applied psychology.

[62]  F. Schmidt,et al.  General Mental Ability, Job Performance, and Red Herrings: Responses to Osterman, Hauser, and Schmitt , 2007 .

[63]  Filip Lievens,et al.  Combining predictors to achieve optimal trade-offs between selection quality and adverse impact. , 2007, The Journal of applied psychology.

[64]  Catherine M. Dalton,et al.  Debunking Myths and Urban Legends About Meta-Analysis , 2011 .

[65]  Kathleen M. Carley,et al.  Simulation modeling in organizational and management research , 2007 .

[66]  Bryan D. Edwards,et al.  Multistage selection strategies: simulating the effects on adverse impact and expected performance for various predictor combinations. , 2009, The Journal of applied psychology.

[67]  Paul R. Sackett,et al.  THE EFFECTS OF FORMING MULTI‐PREDICTOR COMPOSITES ON GROUP DIFFERENCES AND ADVERSE IMPACT , 1997 .

[68]  Richard L. Griffith,et al.  Faking on Personality Measures: Implications for Selection Involving Multiple Predictors , 2009 .

[69]  Herman Aguinis,et al.  Revival of test bias research in preemployment testing. , 2010, The Journal of applied psychology.

[70]  A Monte Carlo Examination of Banding and Rank Order Methods of Test Score Use in Personnel Selection , 1991 .

[71]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[72]  Steven D. Maurer,et al.  The validity of employment interviews: A comprehensive review and meta-analysis. , 1994 .

[73]  Fred S. Switzer,et al.  PRIOR SELECTION CAUSES BIASED ESTIMATES OF STANDARDIZED ETHNIC GROUP DIFFERENCES: SIMULATION AND ANALYSIS , 2001 .

[74]  N. Schmitt The Value of Personnel Selection: Reflections on Some Remarkable Claims , 2007 .

[75]  James E. Campion,et al.  Using the 4/5ths Rule as an Outcome in Regression Analyses: A Demonstrative Simulation , 2008 .

[76]  K. Murphy,et al.  Can Changes in Differential Dropout Rates Reduce Adverse Impact? A Computer Simulation Study of a Multi‐wave Selection System , 2004 .

[77]  Keith Hattrup,et al.  The effects of varying conceptualizations of job performance on adverse impact, minority hiring, and predicted performance. , 1997 .

[78]  R. Landis,et al.  When small effect sizes tell a big story, and when large effect sizes don't. , 2009 .

[79]  J. Steenkamp,et al.  A Review and Evaluation of Meta-Analysis Practices in Management Research , 2009 .

[80]  Herman Aguinis,et al.  Sampling variance in the correlation coefficient under indirect range restriction : Implications for validity generalization , 1997 .

[81]  Anton J. Villado,et al.  The importance of distinguishing between constructs and methods when comparing predictors in personnel selection research and practice. , 2008, The Journal of applied psychology.

[82]  Huy Le,et al.  Increasing the Accuracy of Corrections for Range Restriction: Implications for Selection Procedure Validities and Other Research Results , 2006 .

[83]  F. Schmidt,et al.  A Test of Two Refinements in Procedures for Meta-Analysis , 1994 .

[84]  Paul R. Sackett,et al.  NEW DEVELOPMENTS IN THE USE OF MEASURES OF HONESTY INTEGRITY, CONSCIENTIOUSNESS, DEPENDABILITY TRUSTWORTHINESS, AND RELIABILITY FOR PERSONNEL SELECTION , 1996 .

[85]  Younghwa Lee,et al.  The Technology Acceptance Model: Past, Present, and Future , 2003, Commun. Assoc. Inf. Syst..

[86]  M. Dean On Biodata Construct Validity, Criterion-Related Validity, and Adverse Impact. , 1999 .

[87]  Jill E. Ellingson,et al.  High-stakes testing in employment, credentialing, and higher education. Prospects in a post-affirmative-action world. , 2001, The American psychologist.

[88]  Deniz S. Ones,et al.  GROUP DIFFERENCES IN PERSONALITY: META‐ANALYSES COMPARING FIVE U.S. RACIAL GROUPS , 2008 .

[89]  P. Sackett,et al.  Subgroup differences on cognitively loaded tests in contexts other than personnel selection , 2009 .

[90]  J. Colquitt,et al.  Toward an integrative theory of training motivation: a meta-analytic path analysis of 20 years of research. , 2000, The Journal of applied psychology.

[91]  Winfred Arthur,et al.  Hunter and Hunter (1984) revisited: Interview validity for entry-level jobs. , 1994 .

[92]  Brent A. Scott,et al.  Self-efficacy and work-related performance: the integral role of individual differences. , 2007, The Journal of applied psychology.

[93]  Frank L. Schmidt,et al.  Biographical Data in Employment Selection: Can Validities Be Made Generalizable? , 1990 .

[94]  John P. Wanous,et al.  The role of judgment calls in meta-analysis. , 1989 .

[95]  Michael A. McDaniel,et al.  Subgroup Differences in Situational Judgment Test Performance: A Meta-Analysis , 2008 .