The Use (and Misuse) of Statistical Significance Testing: Some Recommendations for Improved Editorial Policy and Practice.

The paper evaluates the logic underlying various criticisms of statistical significance testing and makes specific recommendations for scientific and editorial practice that might better increase the knowledge base. The effects of contemporary significance testing practice on the literature are evaluated. The paper explores why unconscious preferences for certain practices have emerged and why such practices are so impervious to change. The paper attempts to facilitate escape from some of the methodological paradigms that tend to unconsciously govern thinking regarding the processes of scientific inquiry. Few methodological offerings have sparked more controversy than Sir Ronald Fisher's (1925; 1926) contribution ..... the logic of null hypothesis testing. The last 30 years have involved periodic efforts (cf. Carver, 1978; Morrison & Henkel, 1970; Selvin, 1957;) by various researchers "to exorcise the null hypothesis" (Cronbach, 1975, p. 124). For example, Shaver (1979, pp. 5-6) has argued that The emphasis on statistics and the "test of significance" procedure has resulted in a methodolor:Ical orientation toward establishing generalizability that has ben deleterious in its effects on the scientific accumulation of knowledge

[1]  James P. Shaver,et al.  Randomness and Replication in Ten Years of the American Educational Research Journal , 1980 .

[2]  James F. McNamara Practical Significance and Statistical Models , 1978 .

[3]  Donald R. Atkinson,et al.  Statistical Significance, Power, and Effect Size: A Response to the Reexamination of Reviewer Bias. , 1983 .

[4]  F. Alberoni,et al.  Contribution to the study of subjective probability. I. , 1962, The Journal of general psychology.

[5]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[6]  M. Patton Alternative Evaluation Research Paradigm. , 1975 .

[7]  Jacob Cohen Multiple regression as a general data-analytic system. , 1968 .

[8]  Reuven Dar,et al.  Another look at Meehl, Lakatos, and the scientific practices of psychologists. , 1987 .

[9]  Lee S. Shulman,et al.  Disciplines of Inquiry in Education: An Overview , 1981 .

[10]  Joseph Berkson,et al.  Some Difficulties of Interpretation Encountered in the Application of the Chi-Square Test , 1938 .

[11]  Hanan C. Selvin,et al.  A Critique of Tests of Significance in Survey Research , 1957 .

[12]  W. D. Hudson The Is-Ought Question , 1969 .

[13]  T. R. Knapp Canonical correlation analysis: A general parametric significance-testing system. , 1978 .

[14]  Geoffrey Keppel,et al.  Science and Behavior: An Introduction to Methods of Research. 2nd ed. , 1981 .

[15]  P. Ashton,et al.  Improving Educational Research Through the Development of Educational Paradigms , 1983 .

[16]  K. Strike An Epistemology of Practical Research1 , 1979 .

[17]  Stephen Olejnik,et al.  Planning Educational Research: Determining the Necessary Sample Size. , 1984 .

[18]  G. Loftus Essence Of Statistics , 1982 .

[19]  J. Shaver Readdressing the Role of Statistical Tests of Significance. , 1980 .

[20]  J. Shaver The Productivity of Educational Research and the Applied-Basic Research Distinction1 , 1979 .

[21]  B. J. Winer The Significance Test Controversy--A Reader. , 1971 .

[22]  L. Dalgleish,et al.  Statistical inference in personality research , 1982 .

[23]  J. Tukey,et al.  AVERAGE VALUES OF MEAN SQUARES IN FACTORIALS , 1956 .

[24]  L. Cronbach Beyond the Two Disciplines of Scientific Psychology. , 1975 .

[25]  T. Sterling Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[26]  Robert Rosenthal,et al.  The Interpretation of Levels of Significance by Psychological Researchers , 1963 .

[27]  B. Thompson Canonical Correlation Analysis: Uses and Interpretation , 1984 .

[28]  Jum C. Nunnally,et al.  The Place of Statistics in Psychology , 1960 .

[29]  Leslie Kish,et al.  Some Statistical Problems in Research Design , 1959 .

[30]  Kenneth A. Kavale,et al.  The Efficacy of Special Versus Regular Class Placement for Exceptional Children: a Meta-Analysis , 1980 .

[31]  E. Eisner Anastasia Might Still be Alive, But the Monarchy is Dead , 1983 .

[32]  F. Alberoni Contribution to the study of subjective probability: prediction. II. , 1962, The Journal of general psychology.

[33]  Imre Lakatos,et al.  The Methodology of Scientific Research Programmes , 1978 .

[34]  L. Cohen Clinical psychologists' judgments of the scientific merit and clinical relevance of psychotherapy outcome research. , 1979, Journal of consulting and clinical psychology.

[35]  Glenn L. Rowley,et al.  The Reliability of Observational Measures. , 1976 .

[36]  Frank Yates,et al.  The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics , 1951 .

[37]  Maurice M. Tatsuoka An Examination of the Statistical Properties of a Multivariate Measure of Strength of Relationship. Final Report. , 1973 .

[38]  A. Signorelli Statistics: Tool or master of the psychologist? , 1974 .

[39]  D. Bakan,et al.  The test of significance in psychological research. , 1966, Psychological bulletin.

[40]  W. W. Daniel Statistical significance versus practical significance , 1977 .

[41]  Bruce E. Wampold,et al.  Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship? , 1982 .

[42]  R. McGinnis RANDOMIZATION AND INFERENCE IN SOCIOLOGICAL RESEARCH , 1958 .

[43]  R. Brennan Elements of generalizability theory , 1983 .

[44]  Bruce Thompson,et al.  ANOVA versus Regression Analysis of ATI Designs: An Empirical Investigation , 1986 .

[45]  John M. Neale,et al.  Science and behavior : an introduction to methods of research / John M. Neale , 1973 .

[46]  Bruce Thompson Heuristics for Teaching Multivariate General Linear Model Techniques. , 1985 .

[47]  Anne L. Schneider,et al.  Policy Implications of Using Significance Tests in Evaluation Research , 1984 .

[48]  James R. Craig,et al.  Significance tests and their interpretation: An example utilizing published research and ω2 , 1976 .