Significance Tests Die Hard

We present a critique showing the flawed logical structure of statistical significance tests. We then attempt to analyze why, in spite of this faulty reasoning, the use of significance tests persists. We identify the illusion of probabilistic proof by contradiction as a central stumbling block, because it is based on a misleading generalization of reasoning from logic to inference under uncertainty. We present new data from a student sample and examples from the psychological literature showing the strength and prevalence of this illusion. We identify some intrinsic cognitive mechanisms (similarity to modus tollens reasoning; verbal ambiguity in describing the meaning of significance tests; and the need to rule out chance findings) and extrinsic social pressures which help to maintain the illusion. We conclude by mentioning some alternative methods for presenting and analyzing psychological data, none of which can be considered the ultimate method.

[1]  W. W. Rozeboom.  The fallacy of the null-hypothesis significance test. , 1960, Psychological bulletin.

[2]  S. Milgram.  BEHAVIORAL STUDY OF OBEDIENCE. , 1963, Journal of abnormal psychology.

[3]  D. Bakan,et al.  The test of significance in psychological research. , 1966, Psychological bulletin.

[4]  P. Levy,et al.  Substantive significance of significant differences between two groups. , 1967, Psychological bulletin.

[5]  S. S. Steivens.  Measurement, Statistics, and the Schemapiric View , 1968 .

[6]  S Milgram,et al.  Some Conditions of Obedience and Disobedience to Authority , 1965 .

[7]  H. Friedman.  Magnitude of experimental effect and a table for its rapid estimation. , 1968 .

[8]  D. Lykken.  Statistical significance in psychological research. , 1968, Psychological bulletin.

[9]  L. Guttman.  What is Not What in Statistics , 1977 .

[10]  R. P. Carver.  The Case Against Statistical Significance Testing , 1978 .

[11]  Frederick T. Bacon.  Credibility of Repeated Statements: Memory for Trivia. , 1979 .

[12]  Hypothesis Testing and Proof by Contradiction: An Analogy , 1980 .

[13]  Interpreting Statistical Significance , 1982 .

[14]  D. Eddy.  Judgment under uncertainty: Probabilistic reasoning in clinical medicine: Problems and opportunities , 1982 .

[15]  Statistics and the Law , 1983 .

[16]  M. Bar-Hillel,et al.  Probabilistic Dependence Between Events , 1983 .

[17]  G A Diamond,et al.  Clinical trials and statistical verdicts: probable grounds for appeal. , 1983, Annals of internal medicine.

[18]  Tests of Significance Violate the Rule of Implication , 1984, PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association.

[19]  Ian Begg,et al.  On believing what we remember. , 1985 .

[20]  Louis Guttman,et al.  The Illogic of Statistical Inference for Cumulative Science , 1984 .

[21]  R. Hogarth,et al.  Judging probable cause. , 1986 .

[22]  Reuven Dar,et al.  Another look at Meehl, Lakatos, and the scientific practices of psychologists. , 1987 .

[23]  Alexander Pollatsek,et al.  Understanding conditional probabilities , 1987 .

[24]  P. Pollard,et al.  On the probability of making Type I errors. , 1987 .

[25]  Siu L. Chow,et al.  Significance test or effect size , 1988 .

[26]  Louis Guttman,et al.  Eta, disco, odisco, andF , 1988 .

[27]  S. Zabell.  R. A. Fisher on the history of inverse probability , 1989 .

[28]  R. Rosenthal,et al.  Statistical Procedures and the Justification of Knowledge in Psychological Science , 1989 .

[29]  Jacob Cohen,et al.  THINGS I HAVE LEARNED (SO FAR) , 1990 .

[30]  S. Chow.  Conceptual Rigor versus Practical Impact , 1991 .

[31]  D. Lindley,et al.  The Analysis of Experimental Data: The Appreciation of Tea and Wine , 1993 .