Statistical Significance Tests, Effect Size Reporting and the Vain Pursuit of Pseudo-Objectivity

Two themes are argued in this comment on the use of statistical significance tests. First, effect sizes are an important aspect of results that should be reported. However, 10 empirical studies (some of several different journals) of articles in various disciplines demonstrate that effect sizes are still not usually being reported, notwithstanding the admonitions of the 1994 American Psychological Association (APA) Publication Manual. Second, using statistical significance tests does not (and cannot) make scientists (or their science) objective.

[1]  Carl J. Huberty,et al.  Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses , 1998 .

[2]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[3]  Bruce Thompson,et al.  IN PRAISE OF BRILLIANCE : WHERE THAT PRAISE REALLY BELONGS , 1998 .

[4]  B. Thompson.  Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms , 1996 .

[5]  Daniel H. Robinson,et al.  Research news and Comment: Reflections on Statistical and Substantive Significance, With a Slice of Replication , 1997 .

[6]  Bruce Thompson,et al.  Rejoinder: Editorial Policies Regarding Statistical Significance Tests: Further Comments , 1997 .

[7]  Johanna E. Nilsson,et al.  Statistical Significance Reporting: Current Trends and Uses in MECD. , 1998 .

[8]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[9]  W. Hays,et al.  Statistics (3rd ed.). , 1982 .

[10]  J. Levin,et al.  Reflections on Statistical and Substantive Significance, with a Slice of Replication. , 1997 .

[11]  Bruce Thompson,et al.  Journal Editorial Policies Regarding Statistical Significance Tests: Heat Is to Fire as p Is to Importance , 1999 .

[12]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[13]  Diane Crawford,et al.  Editorial , 2000, CACM.

[14]  Robert W. Frick.  Defending the Statistical Status Quo , 1999 .

[15]  Comment on Significance Testing. , 1998 .

[16]  Bruce Thompson,et al.  The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives , 1993 .

[17]  W. Dunlap,et al.  On the Logic and Purpose of Significance Testing , 1997 .

[18]  B. Thompson.  Editorial Policies Regarding Statistical Significance Testing : Three Suggested Reforms , 2012 .

[19]  B. Thompson,et al.  Further Comments on Statistical Significance Tests. , 1998 .

[20]  Bruce Thompson,et al.  If Statistical Significance Tests are Broken/Misused, What Practices Should Supplement or Replace Them? , 1999 .

[21]  R. Frick,et al.  The appropriate use of null hypothesis testing. , 1996 .

[22]  Carl J. Huberty,et al.  A Single Contrast Test Procedure , 1988 .