The Role of Statistical Significance Testing In Educational Research

The research methodology literature in recent years has included a full frontal assault on statistical significance testing.The purpose of this paper is to promote the position that, while significance testing as the sole basis for resultinterpretation is a fundamentally flawed practice, significance tests can be useful as one of several elements in acomprehensive interpretation of data. Specifically, statistical significance is but one of three criteria that must bedemonstrated to establish a position empirically. Statistical significance merely provides evidence that an event did nothappen by chance. However, it provides no information about the meaningfulness (practical significance) of an eventor if the result is replicable. Thus, we support other researchers who recommend that statistical sign ificance testing mustbe accompanied by judgments of the event’s practical significance and replicability.

[1]  J. Levin.  Statistical Significance Testing From Three Perspectives , 1993 .

[2]  Bruce Thompson,et al.  Rejoinder: Editorial Policies Regarding Statistical Significance Tests: Further Comments , 1997 .

[3]  Bruce Thompson,et al.  The Use (and Misuse) of Statistical Significance Testing: Some Recommendations for Improved Editorial Policy and Practice. , 1987 .

[4]  L. Harlow,et al.  What if there were no significance tests , 1997 .

[5]  L. Cronbach.  Beyond the Two Disciplines of Scientific Psychology. , 1975 .

[6]  R. A. Weitzman,et al.  Seven Treacherous Pitfalls of Statistics, Illustrated , 1984 .

[7]  Anne L. Schneider,et al.  Policy Implications of Using Significance Tests in Evaluation Research , 1984 .

[8]  R. P. Carver.  The Case Against Statistical Significance Testing , 1978 .

[9]  Patricia Snyder,et al.  Use of Tests of Statistical Significance and Other Analytic Choices in a School Psychology Journal: Review of Practices and Suggested Alternatives. , 1998 .

[10]  Carl J. Huberty,et al.  Historical Origins of Statistical Testing Practices: The Treatment of Fisher versus Neyman-Pearson Views in Textbooks. , 1993 .

[11]  Patricia Snyder,et al.  Evaluating Results Using Corrected and Uncorrected Effect Size Estimates , 1993 .

[12]  Bruce Thompson,et al.  The Use of Statistical Significance Tests in Research: Bootstrap and Other Alternatives , 1993 .

[13]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[14]  W. Dunlap,et al.  On the Logic and Purpose of Significance Testing , 1997 .

[15]  Monica J. Harris.  Significance Tests are Not Enough , 1991 .

[16]  Patricia Snyder,et al.  Statistical Significance Testing Practices in The Journal of Experimental Education , 1997 .

[17]  P. Meehl.  Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[18]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[19]  J. Hunter.  Needed: A Ban on the Significance Test , 1997 .

[20]  Bruce Thompson,et al.  WHY ENCOURAGING EFFECT SIZE REPORTING IS NOT WORKING : THE ETIOLOGY OF RESEARCHER RESISTANCE TO CHANGING PRACTICES , 1999 .

[21]  Denton E. Morrison,et al.  The Significance Test Controversy , 1972 .

[22]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[23]  James P. Shaver,et al.  What Statistical Significance Testing Is, and What It Is Not , 1993 .

[24]  H. Suen.  Significance Testing , 1992 .

[25]  Bruce Thompson,et al.  Statistical Significance, Result Importance, and Result Generalizability: Three Noteworthy But Somewhat Different Issues , 1989 .

[26]  R. Frick.  Accepting the null hypothesis , 1995, Memory & cognition.

[27]  M. Masson,et al.  Using confidence intervals in within-subject designs , 1994, Psychonomic bulletin & review.

[28]  R. Falk,et al.  Significance Tests Die Hard , 1995 .

[29]  R. P. Carver.  The Case Against Statistical Significance Testing, Revisited , 1993 .

[30]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[31]  B. Thompson.  Research news and Comment: AERA Editorial Policies Regarding Statistical Significance Testing: Three Suggested Reforms , 1996 .

[32]  J. Levin,et al.  Reflections on Statistical and Substantive Significance, with a Slice of Replication. , 1997 .