Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations.

Although dissatisfaction with the limitations associated with tests for statistical significance has been growing for several decades, applied researchers have continued to rely almost exclusively on these indicators of effect when reporting their findings. To encourage an increased use of alternative measures of effect, the present paper discusses several measures of effect size that might be used in group comparison studies involving univariate and/or multivariate models. For the methods discussed, formulas are presented and data from an experimental study are used to demonstrate the application and interpretation of these indices. The paper concludes with some cautionary notes on the limitations associated with these measures of effect size. Copyright 2000 Academic Press.

[1]  T. L. Kelley,et al.  An Unbiased Correlation Ratio Measure. , 1935, Proceedings of the National Academy of Sciences of the United States of America.

[2]  W. Hays.  Statistics for psychologists , 1963 .

[3]  B. Wolman,et al.  Handbook of clinical psychology , 1965 .

[4]  Misinterpreting the significance of "explained variation." , 1967 .

[5]  H. Friedman.  Magnitude of experimental effect and a table for its rapid estimation. , 1968 .

[6]  Michael C. Corballis,et al.  Beyond tests of significance: Estimating strength of effects in selected ANOVA designs. , 1969 .

[7]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[8]  G. Glass,et al.  Measures of Association in Comparative Experiments: Their Development and Interpretation , 1969 .

[9]  R. Kirk.  Experimental Design: Procedures for the Behavioral Sciences , 1970 .

[10]  C. Huberty.  MULTIVARIATE INDICES OF STRENGTH OF ASSOCIATION. , 1972, Multivariate behavioral research.

[11]  I. Smith.  THE ETA COEFFICIENT IN MANOVA. , 1972, Multivariate behavioral research.

[12]  J. Stevens,et al.  GLOBAL MEASURES OF ASSOCIATION IN MULTIVARIATE ANALYSIS OF VARIANCE. , 1972, Multivariate behavioral research.

[13]  Darshan Sachdeva.  Estimating Strength of Relationship in Multivariate Analysis of Variance , 1973 .

[14]  L. S. Feldt.  What Size Samples for Methods/Materials Experiments? , 1973 .

[15]  D. H. Dodd,et al.  Computational procedures for estimating magnitude of effect for some analysis of variance designs. , 1973 .

[16]  J. Dwyer.  Analysis of variance and the magnitude of effects: A general approach. , 1974 .

[17]  R. Carroll,et al.  Sampling Characteristics of Kelley's ε and Hays' ω , 1975 .

[18]  T. Gerig.  Multivariate Analysis: Techniques for Educational and Psychological Research , 1975 .

[19]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[20]  Practical Significance in Program Evaluation , 1978 .

[21]  James F. McNamara.  Practical Significance and Statistical Models , 1978 .

[22]  R. P. Carver.  The Case Against Statistical Significance Testing , 1978 .

[23]  J. Gaebelein,et al.  The Utility of Within-Subjects Variables: Estimates of Strength , 1978 .

[24]  Donald B. Rubin,et al.  A Note on Percent Variance Explained as A Measure of the Importance of Effects , 1979 .

[25]  Charles Lewis,et al.  Partial Omega Squared for Anova Designs , 1979 .

[26]  W. Alan Nicewander,et al.  Some symmetric, invariant measures of multivariate association , 1979 .

[27]  Lee Sechrest,et al.  Meaningful measures of effect. , 1981 .

[28]  D. P. Hartmann,et al.  A cautionary note on the use of omega squared to evaluate the effectiveness of behavioral treatments , 1981 .

[29]  L. Hedges.  Distribution Theory for Glass's Estimator of Effect size and Related Estimators , 1981 .

[30]  Scott E. Maxwell,et al.  Measures of strength of association: A comparative examination , 1981 .

[31]  G. Glass,et al.  Meta-analysis in social research , 1981 .

[32]  Ronald C. Serlin,et al.  A multivariate measure of association based on the Pillai-Bartlett procedure. , 1982 .

[33]  L. Sechrest,et al.  Magnitudes of Experimental Effects in Social Science Research , 1982 .

[34]  K. O’grady,et al.  Measures of explained variance: Cautions and limitations. , 1982 .

[35]  Donald B. Rubin,et al.  A Simple, General Purpose Display of Magnitude of Experimental Effect , 1982 .

[36]  W. Hays.  Experimental Design: Procedures for the Behavioral Sciences. 2nd ed. , 1983 .

[37]  L. Murray,et al.  How significant is a significant difference? Problems with the measurement of magnitude of effect. , 1987 .

[38]  R. Rosenthal,et al.  Focused Tests of Significance and Effect Size Estimation in Counseling Psychology. , 1988 .

[39]  M. Strube.  Some Comments on the Use of Magnitude-of-Effect Estimates. , 1988 .

[40]  R. Bargmann,et al.  Multivariate Analysis (Techniques for Educational and Psychological Research) , 1989 .

[41]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[42]  Scott E. Maxwell,et al.  Designing Experiments and Analyzing Data: A Model Comparison Perspective , 1990 .

[43]  K. McGraw,et al.  A common language effect size statistic. , 1992 .

[44]  Frank L. Schmidt,et al.  What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. , 1992 .

[45]  James F. Baumann,et al.  Effect of Think-Aloud Instruction on Elementary Students' Comprehension Monitoring Abilities , 1992 .

[46]  Carl J. Huberty,et al.  Preliminary Statistical Tests. , 1993 .

[47]  R. P. Carver.  The Case Against Statistical Significance Testing, Revisited , 1993 .

[48]  R. Rosenthal.  Parametric measures of effect size. , 1994 .

[49]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[50]  S. Brooks,et al.  Applied Multivariate Statistics for the Social Sciences , 1993 .

[51]  Sherrie L. Nist,et al.  The Role of Context and Dictionary Definitions on Varying Levels of Word Knowledge. , 1995 .

[52]  Roger E. Kirk,et al.  Experimental design: Procedures for the behavioral sciences (3rd ed.). , 1995 .

[53]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[54]  C. Heckler.  Applied Discriminant Analysis , 1995 .

[55]  W. Dunlap,et al.  Meta-Analysis of Experiments With Matched Groups or Repeated Measures Designs , 1996 .

[56]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[57]  John T. E. Richardson.  Measures of effect size , 1996 .

[58]  B. Thompson.  Editorial Policies Regarding Statistical Significance Testing : Three Suggested Reforms , 2012 .

[59]  B. Thompson.  Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms , 1996 .

[60]  Kent B. Monroe,et al.  Effect-Size Estimates: Issues and Problems in Interpretation , 1996 .

[61]  Scott B. Morris,et al.  Correcting effect sizes computed from factor analysis of variance for use in meta-analysis. , 1997 .

[62]  Bruce Thompson,et al.  Rejoinder: Editorial Policies Regarding Statistical Significance Tests: Further Comments , 1997 .

[63]  J. Levin,et al.  Reflections on Statistical and Substantive Significance, with a Slice of Replication. , 1997 .

[64]  Daniel H. Robinson,et al.  Research news and Comment: Reflections on Statistical and Substantive Significance, With a Slice of Replication , 1997 .

[65]  J. Hunter.  Needed: A Ban on the Significance Test , 1997 .

[66]  Carl J. Huberty,et al.  Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses , 1998 .

[67]  Bruce Thompson,et al.  If Statistical Significance Tests are Broken/Misused, What Practices Should Supplement or Replace Them? , 1999 .

[68]  Bruce Thompson,et al.  Journal Editorial Policies Regarding Statistical Significance Tests: Heat Is to Fire as p Is to Importance , 1999 .

[69]  Daniel H. Robinson,et al.  Further Reflections on Hypothesis Testing and Editorial Policy for Primary Research Journals , 1999 .

[70]  Robert W. Frick.  Defending the Statistical Status Quo , 1999 .

[71]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .