Meta-Analysis Calculations Based on Independent and Nonindependent Cases

Meta-analysis calculations, including both validity generalization and reliability generalization, are based on the assumption that the correlations or standardized mean differences are statistically independent. This assumption is frequently violated because studies often report more than one correlation or effect size based on the same sample. The purpose of this study was to determine the effect of including nonindependent correlations in Hunter and Schmidt’s meta-analysis method on the estimated population standard deviation. First, the method for estimating the population standard deviation was examined and four new estimation methods were suggested and evaluated. The evaluation indicated that the Hunter and Schmidt method will underestimate the true population standard deviation, and the new methods were developed to correct for this. Second, a pseudo–Monte Carlo study consisting of 36 different simulated cases was conducted to illustrate the behavior of both the Hunter and Schmidt estimators and the new methods.

[1]  Larry V. Hedges,et al.  An unbiased correction for sampling error in validity generalization studies. , 1989 .

[2]  Development and Monte Carlo Evaluation of Meta-Analytic Estimators for Correlated Data , 1991 .

[3]  M Martinussen,et al.  Psychological measures as predictors of pilot performance: a meta-analysis. , 1996, The International journal of aviation psychology.

[4]  M. Koslowsky,et al.  COMPONENTS OF ARTIFACTUAL VARIANCE IN META‐ANALYTIC RESEARCH , 1994 .

[5]  John E. Hunter,et al.  Development of a general solution to the problem of validity generalization. , 1977 .

[6]  W. R. Buckland,et al.  Distributions in Statistics: Continuous Multivariate Distributions , 1974 .

[7]  Donald B. Rubin,et al.  Meta-Analytic Procedures for Combining Studies With Multiple Effect Sizes , 1986 .

[8]  T. Vacha-Haase,et al.  Reliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies , 1998 .

[9]  R. Ladd,et al.  Power and Accuracy of the Schmidt and Hunter Meta-Analytic Procedures , 1993 .

[10]  Patricia B. Elmore,et al.  Correlational Meta-Analysis: Independent and Nonindependent Cases , 1992 .

[11]  W. R. Buckland,et al.  Distributions in Statistics: Continuous Multivariate Distributions , 1973 .

[12]  L. A. Marascuilo,et al.  Large Sample Multivariate Procedures for Comparing and Combining Effect Sizes Within a Single Study , 1987 .

[13]  R. Rosenthal.  Meta-analytic procedures for social research , 1984 .

[14]  Robert L. Bangert-Drowns,et al.  Review of developments in meta-analytic method. , 1986 .

[15]  Gregg B. Jackson,et al.  Meta-Analysis: Cumulating Research Findings Across Studies , 1982 .

[16]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .