Implications of direct and indirect range restriction for meta-analysis methods and findings.

Range restriction in most data sets is indirect, but the meta-analysis methods used to date have applied the correction for direct range restriction to data in which range restriction is indirect. The authors show that this results in substantial undercorrections for the effects of range restriction, and they present meta-analysis methods for making accurate corrections when range restriction is indirect. Applying these methods to a well-known large-sample empirical database, the authors estimate that previous meta-analyses have underestimated the correlation between general mental ability and job performance by about 25%, indicating that this is potentially an important methodological issue in meta-analysis in general.

[1]  Huy Le,et al.  Increasing the Accuracy of Corrections for Range Restriction: Implications for Selection Procedure Validities and Other Research Results , 2006 .

[2]  John E. Hunter,et al.  Dichotomization of continuous variables: the implications for meta-analysis , 1990 .

[3]  Malcolm James Ree,et al.  Rangej: A Pascal Program to Compute the Multivariate Correction for Range Restriction , 1994 .

[4]  John E. Hunter,et al.  Task Differences as Moderators of Aptitude Test Validity in Selection: A Red Herring , 1981 .

[5]  J. Mendoza,et al.  Large-sample confidence intervals for validity and reliability coefficients. , 2000, Psychological methods.

[6]  John E. Hunter,et al.  Impact of valid selection procedures on work-force productivity. , 1979 .

[7]  John E. Hunter,et al.  Development of a general solution to the problem of validity generalization. , 1977 .

[8]  John E. Hunter,et al.  Theory Testing and Measurement Error. , 1999 .

[9]  F. Schmidt,et al.  A Test of Two Refinements in Procedures for Meta-Analysis , 1994 .

[10]  D. Lawley.  IV.—A Note on Karl Pearson's Selection Formulæ , 1944, Proceedings of the Royal Society of Edinburgh. Section A. Mathematical and Physical Sciences.

[11]  Fred S. Switzer,et al.  PRIOR SELECTION CAUSES BIASED ESTIMATES OF STANDARDIZED ETHNIC GROUP DIFFERENCES: SIMULATION AND ANALYSIS , 2001 .

[12]  L. Hedges,et al.  Fixed- and random-effects models in meta-analysis. , 1998 .

[13]  Hubert E. Brogden,et al.  When Testing Pays Off , 1949 .

[14]  Huy Le,et al.  Beyond alpha: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. , 2003, Psychological methods.

[15]  Delwyn L. Harnisch,et al.  Corrections for range restriction: An empirical investigation of conditions resulting in conservative corrections. , 1981 .

[16]  Michael J. Burke,et al.  Two new procedures for studying validity generalization. , 1983 .

[17]  Michael A. McDaniel,et al.  Refinements in validity generalization methods: Implications for the situational specificity hypothesis. , 1993 .

[18]  Michael D. Mumford,et al.  Corrections for Attenuation and Range Restriction on the Predictor , 1987 .

[19]  J. Hunter,et al.  Validity and Utility of Alternative Predictors of Job Performance , 1984 .

[20]  F. Schmidt,et al.  The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. , 1998 .

[21]  F. Schmidt,et al.  Measurement Error in Psychological Research: Lessons From 26 Research Scenarios , 1996 .

[22]  W. Coward,et al.  Linearity of ability-performance relationships: A reconfirmation. , 1990 .

[23]  Karl Pearson,et al.  Mathematical Contributions to the Theory of Evolution. XIX. Second Supplement to a Memoir on Skew Variation , 1901 .

[24]  H. G. Osburn,et al.  Development and test of a new model for validity generalization. , 1980 .

[25]  Cidambi Srinivasan,et al.  Combining Information: Statistical Issues and Opportunities for Research , 1993 .

[26]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[27]  The proper sequence for correcting correlation coefficients for range restriction and unreliability , 2001 .

[28]  Kristopher J Preacher,et al.  On the practice of dichotomization of quantitative variables. , 2002, Psychological methods.

[29]  Brenda White,et al.  DEPARTMENT OF LABOR , 2006 .

[30]  P. Sackett,et al.  Job-specific applicant pools and national norms for cognitive ability tests: implications for range restriction corrections in validation research. , 1994, The Journal of applied psychology.

[31]  Michael J. Strube,et al.  Validity Generalization: A Critical Review , 2004 .

[32]  John E. Hunter,et al.  Validity generalization results for computer programmers. , 1980 .

[33]  Gregg B. Jackson,et al.  Meta-Analysis: Cumulating Research Findings Across Studies , 1982 .

[34]  Thomas R. Carretta,et al.  Sign Changes When Correcting for Range Restriction: A Note on Pearson's and Lawley's Selection Formulas , 1994 .

[35]  John E. Hunter,et al.  FORTY QUESTIONS ABOUT VALIDITY GENERALIZATION AND META‐ANALYSIS , 1985 .

[36]  J W Hedge,et al.  Personnel selection. , 1997, Annual review of psychology.

[37]  D. P. Gaver,et al.  Combining Information: Statistical Issues and Opportunities for Research , 1993 .

[38]  P. Sackett,et al.  Correction for range restriction: an expanded typology. , 2000, The Journal of applied psychology.

[39]  Karl Pearson,et al.  Mathematical Contributions to the Theory of Evolution. XI. On the Influence of Natural Selection on the Variability and Correlation of Organs , 1903 .

[40]  F. Schmidt,et al.  Nonlinearity of Range Corrections in Meta-Analysis: Test of an Improved Procedure , 1994 .

[41]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[42]  John E. Hunter,et al.  Fixed Effects vs. Random Effects Meta‐Analysis Models: Implications for Cumulative Research Knowledge , 2000 .

[43]  A. Field.  Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects methods. , 2001, Psychological methods.

[44]  Chockalingam Viswesvaran,et al.  Comparative analysis of the reliability of job performance ratings , 1996 .

[45]  F. Schmidt,et al.  History, Development, Evolution, and Impact of Validity Generalization and Meta-Analysis Methods, 1975–2001 , 2003 .

[46]  John E. Hunter,et al.  Statistical power in criterion-related validation studies. , 1976 .

[47]  Michael T Brannick,et al.  Comparison of two random-effects methods of meta-analysis. , 2002, The Journal of applied psychology.

[48]  Michael J. Burke,et al.  A new meta-analytic approach. , 1991 .