Standardized or simple effect size: what should be reported?

It is regarded as best practice for psychologists to report effect size when disseminating quantitative research findings. Reporting of effect size in the psychological literature is patchy - though this may be changing - and when reported it is far from clear that appropriate effect size statistics are employed. This paper considers the practice of reporting point estimates of standardized effect size and explores factors such as reliability, range restriction and differences in design that distort standardized effect size unless suitable corrections are employed. For most purposes simple (unstandardized) effect size is more robust and versatile than standardized effect size. Guidelines for deciding what effect size metric to use and how to report it are outlined. Foremost among these are: (i) a preference for simple effect size over standardized effect size, and (ii) the use of confidence intervals to indicate a plausible range of values the effect might take. Deciding on the appropriate effect size statistic to report always requires careful thought and should be influenced by the goals of the researcher, the context of the research and the potential needs of readers.

[1]  G. Cumming,et al.  Statistical Reform in Psychology , 2007, Psychological science.

[2]  R. McGrath,et al.  When effect sizes disagree: the case of r and d. , 2006, Psychological methods.

[3]  Huy Le,et al.  Implications of direct and indirect range restriction for meta-analysis methods and findings. , 2006, The Journal of applied psychology.

[4]  Thomas R. Carretta,et al.  The Role of Measurement Error in Familiar Statistics , 2006 .

[5]  Kristopher J Preacher,et al.  Use of the extreme groups approach: a critical reexamination and new recommendations. , 2005, Psychological methods.

[6]  P. Warr,et al.  Copyright © The British Psychological Society Unauthorised use and reproduction in any form (including the internet and other electronic means) is prohibited without prior permission from the Society. , 2005 .

[7]  Michael J. Strube,et al.  Validity Generalization: A Critical Review , 2004 .

[8]  Thom Baguley,et al.  Understanding statistical power in the context of applied research. , 2004, Applied ergonomics.

[9]  A. Buchner,et al.  Auditory Negative Priming in Younger and Older Adults , 2004, The Quarterly journal of experimental psychology. A, Human experimental psychology.

[10]  F Dan Richard,et al.  Meta-analysis of raw mean differences. , 2003, Psychological methods.

[11]  J. Algina,et al.  Generalized eta and omega squared statistics: measures of effect size for some common research designs. , 2003, Psychological methods.

[12]  Raphael Gillett,et al.  The metric comparability of meta-analytic effect-size estimators from factorial designs. , 2003, Psychological methods.

[13]  Tiffany A. Whittaker,et al.  It's Not Effect Sizes So Much as Comments About Their Magnitude That Mislead Readers , 2003 .

[14]  R. DeShon.  A generalizability theory perspective on measurement error corrections in validity generalization. , 2003 .

[15]  Fiona Fidler,et al.  The Fifth edition of the Apa Publication Manual: Why its Statistics Recommendations are so Controversial , 2002 .

[16]  H. Pashler.  STEVENS' HANDBOOK OF EXPERIMENTAL PSYCHOLOGY , 2002 .

[17]  G. Loftus.  Analysis, Interpretation, and Visual Presentation of Experimental Data , 2002 .

[18]  R. DeShon,et al.  Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. , 2002, Psychological methods.

[19]  Russell V. Lenth,et al.  Some Practical Guidelines for Effective Sample Size Determination , 2001 .

[20]  Philip L. Roth,et al.  Correcting the Effect Size of d for Range Restriction and Unreliability , 2001 .

[21]  Stephen J. Payne,et al.  Given-New Versus New-Given?: An analysis of reading times for spatial descriptions , 2000 .

[22]  S D Walter,et al.  Choice of effect measure for epidemiological data. , 2000, Journal of clinical epidemiology.

[23]  Olejnik,et al.  Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations. , 2000, Contemporary educational psychology.

[24]  J. Raaijmakers,et al.  How to deal with "The language-as-fixed-effect fallacy": Common misconceptions and alternative solutions. , 1999 .

[25]  John E. Hunter,et al.  Theory Testing and Measurement Error. , 1999 .

[26]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[27]  Robert W. Frick.  Defending the Statistical Status Quo , 1999 .

[28]  Jacob Cohen,et al.  The problem of units and the circumstance for POMP , 1999 .

[29]  Mark Fichman,et al.  Variance Explained: Why Size Does Not (Always) Matter , 1999 .

[30]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[31]  W. Dunlap,et al.  Meta-Analysis of Experiments With Matched Groups or Repeated Measures Designs , 1996 .

[32]  L. Hedges,et al.  The Handbook of Research Synthesis , 1995 .

[33]  R. Abelson.  Statistics As Principled Argument , 1995 .

[34]  Joseph L. Fleiss,et al.  Measures of effect size for categorical data. , 1994 .

[35]  R. Rosenthal.  Parametric measures of effect size. , 1994 .

[36]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[37]  S Greenland,et al.  The fallacy of employing standardized regression coefficients and correlations as measures of effect. , 1986, American journal of epidemiology.

[38]  K. O’grady,et al.  Measures of explained variance: Cautions and limitations. , 1982 .

[39]  Jae-on Kim,et al.  Standardization in Causal Analysis , 1981 .

[40]  G. Glass,et al.  Meta-analysis in social research , 1981 .

[41]  H. H. Clark.  The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. , 1973 .

[42]  John W. Tukey,et al.  Analyzing data: Sanctification or detective work? , 1969 .

[43]  E. Ghiselli.  Theory of psychological measurement , 1964 .