Improving Research Clarity and Usefulness with Effect Size Indices as Supplements to Statistical Significance Tests

Articles published in professional journals are expected to comply with the 1994 publication manual of the American Psychological Association, which acknowledges that p values are not acceptable measures of effect size magnitudes, and explicitly encourages authors to report effect sizes. This “encouragement” is also consistent with recent criticisms of common misuses of statistical significance tests. The present article examined effect size reporting in 23 quantitative articles reported in Exceptional Children. It was found that effect sizes are rarely being reported, though exemplary reporting practices were also noted Reasons why the APA “encouragement” has been ineffective are explored and it is suggested that an editorial policy requiring effect size reporting similar to that invoked at several journals, including some APA journals, may be useful.

[1]  Robert Rosenthal,et al.  The Interpretation of Levels of Significance by Psychological Researchers , 1963 .

[2]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[3]  Edward L. Vockell,et al.  Perceptions of Document Quality and Use by Educational Decision Makers and Researchers 1 , 1974 .

[4]  Bruce W. Hall,et al.  Evaluation of Published Educational Research: A National Survey1 , 1975 .

[5]  T. R. Knapp.  Canonical correlation analysis: A general parametric significance-testing system. , 1978 .

[6]  F. Borgen,et al.  Uses of discriminant analysis following MANOVA: Multivariate statistics for multivariate purposes. , 1978 .

[7]  Gene V. Glass,et al.  Policy for the Unpredictable (Uncertainty Research and Policy) , 1979 .

[8]  Herbert Friedman,et al.  Simplified Determinations of Statistical Power, Magnitude of Effect and Research Sample Sizes , 1982 .

[9]  Stephen Olejnik,et al.  Planning Educational Research: Determining the Necessary Sample Size. , 1984 .

[10]  Chance and Nonsense: A Conversation about Interpreting Tests of Statistical Significance, Part 1. , 1985 .

[11]  Robert Rosenthal,et al.  Interpretation of significance levels and effect sizes by psychological researchers. , 1986 .

[12]  Bruce W. Hall,et al.  Published Educational Research: An Empirical Study of Its Quality , 1988 .

[13]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[14]  J. Plaza,et al.  The Council for Exceptional Children , 1989 .

[15]  Bruce W. Tuckman,et al.  A Proposal for Improving the Quality of Published Educational Research , 1990 .

[16]  Bruce W. Tuckman,et al.  Research news and Comment: A Proposal for Improving the Quality of Published Educational Research , 1990 .

[17]  Research and Education Reform: Roles for the Office of Educational Research and Improvement. , 1992 .

[18]  B. Thompson.  Two and One‐Half Decades of Leadership in Measurement and Evaluation , 1992 .

[19]  Robert Rosenthal,et al.  Contemporary Issues in the Analysis of Data: A Survey of 551 Psychologists , 1993 .

[20]  Bruce Thompson,et al.  The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives , 1993 .

[21]  Patricia Snyder,et al.  Evaluating Results Using Corrected and Uncorrected Effect Size Estimates , 1993 .

[22]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[23]  Bruce Thompson,et al.  Why Multivariate Methods Are Usually Vital in Research: Some Basic Concepts. , 1994 .

[24]  Bruce Thompson,et al.  Stepwise Regression and Stepwise Discriminant Analysis Need Not Apply here: A Guidelines Editorial , 1995 .

[25]  C. Heckler.  Applied Discriminant Analysis , 1995 .

[26]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[27]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[28]  C. Hanley-Maxwell,et al.  Packing the Parachute: Parents' Experiences as Their Children Prepare to Leave High School , 1996 .

[29]  B. Thompson.  Editorial Policies Regarding Statistical Significance Testing : Three Suggested Reforms , 2012 .

[30]  B. Thompson.  Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms , 1996 .

[31]  Walter R. Borg,et al.  Educational research: An introduction, 6th ed. , 1996 .

[32]  Bonnie Doren,et al.  School-to-Work Components That Predict Postschool Success for Students with and without Disabilities , 1997 .

[33]  M. Wenz-Gross,et al.  Importance of Social Support in the Adjustment of Children with Learning Problems , 1997 .

[34]  S. Odom,et al.  Promoting Positive Attitudes of Kindergarten-Age Children toward People with Disabilities , 1997 .

[35]  M. Pomplun.  When Students with Disabilities Participate in Cooperative Groups , 1997 .

[36]  Joanne Curry Sontag.  Contextual Factors Influencing the Sociability of Preschool Children with Disabilities in Integrated and Segregated Classrooms , 1997 .

[37]  D. Bruns,et al.  Putting Inclusion into Practice: Perspectives of Teachers and Parents , 1997 .

[38]  Response to Bridging the Research-to-Practice Gap , 1997 .

[39]  Early Interventionists Perspectives of Multicultural Practices with African-American Families , 1997 .

[40]  Bruce Thompson,et al.  Five Methodology Errors in Educational Research: The Pantheon of Statistical Significance and Other Faux Pas. , 1998 .

[41]  Carl J. Huberty,et al.  Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses , 1998 .

[42]  Patricia Snyder,et al.  Statistical Significance and Reliability Analyses in Recent Journal of Counseling & Development Research Articles , 1998 .

[43]  Patricia Snyder,et al.  Use of Tests of Statistical Significance and Other Analytic Choices in a School Psychology Journal: Review of Practices and Suggested Alternatives. , 1998 .

[44]  Sharon Vaughn,et al.  Broken Promises: Reading Instruction in the Resource Room , 1998 .

[45]  W. Tryon.  THE INSCRUTABLE NULL HYPOTHESIS , 1998 .

[46]  Lise Saint-Laurent,et al.  Academic Achievement Effects of an in-Class Service Model on Students with and without Disabilities , 1998 .

[47]  Bruce Thompson,et al.  Book Reviews , 1998 .

[48]  Bruce Thompson,et al.  If Statistical Significance Tests are Broken/Misused, What Practices Should Supplement or Replace Them? , 1999 .

[49]  Bruce Thompson,et al.  Journal Editorial Policies Regarding Statistical Significance Tests: Heat Is to Fire as p Is to Importance , 1999 .