Journal Editorial Policies Regarding Statistical Significance Tests: Heat Is to Fire as p Is to Importance

The present paper responds to defenses of statistical significance testing offered by Levin and Robinson. First, some inaccurate perceptions of contemporary criticisms of statistical tests are noted. Second, areas of disagreement are explored. For example, it is noted that all nine empirical studies of reporting practices since 1994 show that “encouraging” (per the 1994 APA style manual) authors to report effect sizes has not worked; two reasons for this failure are explored. Finally, two important areas of agreement regarding needed improvements in contemporary practices are noted.

[1]  Joel R. Levin.  Overcoming feelings of powerlessness in "aging" researchers: a primer on statistical power in analysis of variance designs. , 1997 .

[2]  Carl J. Huberty,et al.  Historical Origins of Statistical Testing Practices: The Treatment of Fisher versus Neyman-Pearson Views in Textbooks. , 1993 .

[3]  R. P. Carver.  The Case Against Statistical Significance Testing, Revisited , 1993 .

[4]  Bruce Thompson,et al.  Five Methodology Errors in Educational Research: The Pantheon of Statistical Significance and Other Faux Pas. , 1998 .

[5]  Bruce Thompson,et al.  Program Facstrap: A Program that Computes Bootstrap Estimates of Factor Structure , 1988 .

[6]  Bruce Thompson,et al.  The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives , 1993 .

[7]  Bruce Thompson,et al.  Discstra: A Computer Program that Computes Bootstrap Resampling Estimates of Descriptive Discriminant Analysis Function and Structure Coefficients and Group Centroids , 1992 .

[8]  Patricia Snyder,et al.  Use of Tests of Statistical Significance and Other Analytic Choices in a School Psychology Journal: Review of Practices and Suggested Alternatives. , 1998 .

[9]  Chance and Nonsense: A Conversation about Interpreting Tests of Statistical Significance, Part 1. , 1985 .

[10]  Patricia Snyder,et al.  Statistical Significance Testing Practices in The Journal of Experimental Education , 1997 .

[11]  M. Oakes.  Statistical Inference: A Commentary for the Social and Behavioural Sciences , 1986 .

[12]  Robert Rosenthal,et al.  The Interpretation of Levels of Significance by Psychological Researchers , 1963 .

[13]  B. Thompson.  Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms , 1996 .

[14]  J. Levin,et al.  Reflections on Statistical and Substantive Significance, with a Slice of Replication. , 1997 .

[15]  Comment on Significance Testing. , 1998 .

[16]  R. Rosenthal.  The file drawer problem and tolerance for null results , 1979 .

[17]  Bruce Thompson,et al.  Exploring the Replicability of a Study's Results: Bootstrap Statistics for the Multivariate Case , 1995 .

[18]  P. Meehl.  Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. , 1978 .

[19]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[20]  Daniel H. Robinson,et al.  Further Reflections on Hypothesis Testing and Editorial Policy for Primary Research Journals , 1999 .

[21]  W. W. Rozeboom.  The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[22]  R. P. Carver.  The Case Against Statistical Significance Testing , 1978 .

[23]  Bruce Thompson,et al.  The pivotal role of replication in psychological research , 1994 .

[24]  Bruce Thompson,et al.  Rejoinder: Editorial Policies Regarding Statistical Significance Tests: Further Comments , 1997 .

[25]  Johanna E. Nilsson,et al.  Statistical Significance Reporting: Current Trends and Uses in MECD. , 1998 .

[26]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[27]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[28]  T. Vacha-Haase,et al.  Reliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies , 1998 .

[29]  B. Thompson,et al.  Further Comments on Statistical Significance Tests. , 1998 .

[30]  Robert Rosenthal,et al.  Interpretation of significance levels and effect sizes by psychological researchers. , 1986 .

[31]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[32]  Bruce Thompson,et al.  IN PRAISE OF BRILLIANCE : WHERE THAT PRAISE REALLY BELONGS , 1998 .

[33]  Robert Rosenthal,et al.  Contemporary Issues in the Analysis of Data: A Survey of 551 Psychologists , 1993 .

[34]  Bruce Thompson,et al.  Statistical Significance, Result Importance, and Result Generalizability: Three Noteworthy But Somewhat Different Issues , 1989 .