Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences

Quantitative research synthesis usually involves the combination of estimates of the standardized mean difference (effect size) derived from independent research studies. In some cases, effect size estimates are available only if the difference between experimental and control group means is statistically significant. If the quantitative result of a study is observed only when the mean difference is statistically significant, the observed mean difference, variance, and effect size are biased estimators of the corresponding population parameters. The exact distribution of the sample effect size is derived for the case in which only studies yielding statistically significant results may be observed. The maximum likelihood estimator of effect size also is derived under the model in which only significant results are observed. The exact distribution of the maximum likelihood estimator is obtained numerically and is used to study the bias of the maximum likelihood estimator. An empirical sampling study is used to supplement the analytic results.

[1]  A. Greenwald.  Consequences of Prejudice Against the Null Hypothesis , 1975 .

[2]  David M. Lane,et al.  Estimating effect size: Bias resulting from the significance criterion in editorial decisions , 1978 .

[3]  Michael J. Strube,et al.  Meta-Analysis and Cross-Cultural Comparison , 1981 .

[4]  Ingram Olkin,et al.  Regression Models in Research Synthesis , 1983 .

[5]  Helena C. Kraemer,et al.  A nonparametric technique for meta-analysis effect size calculation. , 1982 .

[6]  Larry V. Hedges,et al.  A random effects model for effect sizes , 1983 .

[7]  L. J. Chase,et al.  A statistical power analysis of applied psychological research. , 1976 .

[8]  L. Hedges.  Estimation of effect size from a series of independent experiments. , 1982 .

[9]  Larry V. Hedges,et al.  Fitting Continuous Models to Effect Size Data , 1982 .

[10]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[11]  Ralph R. Roberts,et al.  Signifying significant significance. , 1972 .

[12]  G. Glass.  9: Integrating Findings: The Meta-Analysis of Research , 1977 .

[13]  L. Hedges.  Fitting Categorical Models to Effect Sizes from a Series of Experiments , 1982 .

[14]  J. L. Walsh,et al.  The theory of splines and their applications , 1969 .

[15]  D. Bakan,et al.  The test of significance in psychological research. , 1966, Psychological bulletin.

[16]  M. Sidman,et al.  Tactics of scientific research : evaluating experimental data in psychology , 1961 .

[17]  D. Rubin,et al.  Comparing Effect Sizes of Independent Studies , 1982 .

[18]  Larry V. Hedges,et al.  The empirical evidence on the effectiveness of open education , 1980 .

[19]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[20]  L. Hedges.  Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[21]  L. Hedges,et al.  Vote-counting methods in research synthesis. , 1980 .

[22]  R. Senf.  Tactics of Scientific Research: Evaluating Experimental Data in Psychology , 1962 .

[23]  Victor Chew,et al.  Point Estimation of the Parameter of the Binomial Distribution , 1971 .

[24]  Garrett Birkhoff.  The Theory of Splines and Their Applications (J. H. Ahlberg, E. N. Nilson and J. L. Walsh) , 1969 .

[25]  M. J. Alexander,et al.  NEW TABLES OF THE NONCENTRAL T DISTRIBUTION , 1963 .

[26]  A. Eagly,et al.  Sex of researchers and sex-typed communications as determinants of sex differences in influenceability: A meta-analysis of social influence studies , 1981 .