On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients : Some initial Monte Carlo findings

Each of several Monte Carlo simulations generated 100 sets of observed study correlations based on normal, heteroscedastic, or slightly nonlinear bivariate distributions, with one population correlation coefficient and true variance of 0. A version of J. E. Hunter and F. L. Schmidt's (1990b) meta-analysis was applied to each study set. Within simulations, ρ was accurate on average. σ ρ 2 was biased; one would correctly conclude more than half the time that no moderator effects existed. However, cases of variation in ρ and especially in σ ρ 2 indicated that results from individual meta-analyses could deviate substantially from what was found on average. Findings for these no-moderator cases offer applied psychologists some guidelines and cautions when drawing conclusions about true population correlations and true moderator effects (e.g., situational specificity, validity generalization) from meta-analytic results.

[1]  Paul W. Paese,et al.  Validity generalization and hypothetical reliability distributions: A test of the Schmidt-Hunter procedure. , 1988 .

[2]  A. Dalessio,et al.  Stability of validity indices through test score ranges. , 1988 .

[3]  Alice N. Outerbridge,et al.  Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance , 1986 .

[4]  John E. Hunter,et al.  Dichotomization of continuous variables: the implications for meta-analysis , 1990 .

[5]  R. D. Bock,et al.  A multivariate correction for attenuation , 1975 .

[6]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[7]  W. Coward,et al.  Linearity of ability-performance relationships: A reconfirmation. , 1990 .

[8]  William R. Shadish,et al.  Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues. , 1996 .

[9]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[10]  F. Schmidt,et al.  A Test of Two Refinements in Procedures for Meta-Analysis , 1994 .

[11]  Michael J. Burke,et al.  A new meta-analytic approach. , 1991 .

[12]  H. G. Osburn,et al.  Accuracy of Corrections for Restriction in Range Due to Explicit Selection in Heteroscedastic and Nonlinear Distributions , 1980 .

[13]  S. Premack,et al.  Individual unionization decisions. , 1988 .

[14]  John T. Pohlmann,et al.  Fiedler's contingency theory of leadership: an application of the meta-analysis procedures of Schmidt and Hunter , 1985 .

[15]  N. Raju,et al.  An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization. , 1989 .

[16]  Raymond Lee,et al.  Is the validity of a test constant throughout the test score range , 1986 .

[17]  Mark A. Mone,et al.  THE PERCEPTIONS AND USAGE OF STATISTICAL POWER IN APPLIED PSYCHOLOGY AND MANAGEMENT RESEARCH , 1996 .

[18]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[19]  H. G. Osburn,et al.  Multiplicative validity generalization model: Accuracy of estimates as a function of sample size and mean, variance, and shape of distribution of true validities. , 1982 .

[20]  A. Gross,et al.  The Correction for Restriction of Range and Nonlinear Regressions: An Analytic Study , 1987 .

[21]  M. Koslowsky,et al.  COMPONENTS OF ARTIFACTUAL VARIANCE IN META‐ANALYTIC RESEARCH , 1994 .

[22]  Herman Aguinis,et al.  Sampling variance in the correlation coefficient under indirect range restriction : Implications for validity generalization , 1997 .

[23]  H. G. Osburn,et al.  A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization. , 1992 .

[24]  J. Mendoza,et al.  Validity generalization procedures using sample-based estimates: A comparison of six procedures. , 1991 .

[25]  A. Gross,et al.  Restriction of Range Corrections When Both Distribution and Selection Assumptions Are Violated , 1983 .

[26]  Kevin R. Murphy,et al.  Individual differences and behavior in organizations , 1996 .

[27]  A. Gross.  Relaxing the Assumptions Underlying Corrections for Restriction of Range , 1982 .

[28]  John E. Hunter,et al.  Validity generalization results for tests used to predict job proficiency and training success in clerical occupations. , 1980 .

[29]  Fanny Caranikas-Walker,et al.  A meta-analytical structural equations analysis of a model of employee turnover. , 1992 .

[30]  R. Ladd,et al.  Power and Accuracy of the Schmidt and Hunter Meta-Analytic Procedures , 1993 .

[31]  Development and Monte Carlo Evaluation of Meta-Analytic Estimators for Correlated Data , 1991 .

[32]  Gideon Keren,et al.  Statistical and methodological issues in psychology and social sciences research , 1982 .

[33]  Fritz Drasgow,et al.  Bootstrap estimates of standard errors in validity generalization , 1992 .

[34]  R. Rosenthal,et al.  Mediation of interpersonal expectancy effects: 31 meta-analyses. , 1985 .

[35]  Robert L. Linn,et al.  Range restriction problems in the use of self-selected groups for test validation. , 1968 .

[36]  Ellen M. Whitener,et al.  Confusion of confidence intervals and credibility intervals in meta-analysis. , 1990 .

[37]  Paul E. Spector,et al.  Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors. , 1987 .

[38]  Steven P. Brown,et al.  ANTECEDENTS AND CONSEQUENCES OF SALESPERSON JOB SATISFACTION: META ANALYSIS AND ASSESSMENT OF CAUSAL EFFECTS , 1993 .

[39]  P. Bobko.  An analysis of correlations corrected for attenuation and range restriction. , 1983 .

[40]  T. Micceri.  The unicorn, the normal curve, and other improbable creatures. , 1989 .

[41]  Edward R. Kemery,et al.  The power of the Schmidt and Hunter Additive Model of Validity Generalization. , 1987 .

[42]  E. Ghiselli,et al.  The validity of occupational aptitude tests , 1967 .

[43]  Meni Koslowsky,et al.  DETECTING MODERATORS WITH META‐ANALYSIS: AN EVALUATION AND COMPARISON OF TECHNIQUES , 1993 .

[44]  D. Ones,et al.  Theory testing: Combining psychometric meta-analysis and structural equations modeling , 1995 .

[45]  H. Wainer.  Robust Statistics: A Survey and Some Prescriptions , 1976 .

[46]  J. Fisher.  The twisted pear and the prediction of behavior. , 1959 .

[47]  Paul R. Sackett,et al.  On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error. , 1986 .