The Trouble with Significance Tests and What We Can Do About It

ALBERT SPAULDING (1953) pointed out that statistical analysis can present data in thought-provoking ways. Since the time when he made this statement, statistical inference has become commonplace in archaeology. George Cowgill attempts to clarify some aspects of the reasoning behind the use of procedures of statistical inference in archaeology. He has done more than think about statistically presented data, for he looks behind the statistics to their logical underpinnings. There is a gap between the many elementary statistics texts and works on the philosophy of science, a gap that neglects the logic of statistical inference. Most archaeologists labor under considerable misconceptions about the interpretation of statistical results and make serious mistakes as a result. This important paper is generally critical of significance testing as commonly used by archaeologists. Cowgill asks if people are using significance tests correctly, indeed if they know what they are really all about. He also discusses some alternative statistical techniques that may sometimes be of even greater use. The result is an elegant and useful essay on statistical method in archaeology that will help many of us present our data more effectively, and to think about it more precisely. And that is what Spaulding's original concern was in the first place. The rationale of customary "null hypothesis testing" procedures of statistical inference is examined. This approach is not incorrect, but it is prone to misuse and misinterpretation, including neglect of "power" and inappropriate conclusions based on conventional significance levels. The estimation approach, which often seems preferable, is briefly described. The kind of reasoning involved in statistical inference is required whenever we wish to assess the evidence relevant for or against any general proposition, whether we nake any formal computations or not, and whether or not we have observed all possible real instances of relevant evidence. Statistical inference is logically unproblematic if we interpret it as a way of assessing the evidence more clearly. But statistical results cannot be directly converted into probabilities of the truth of hypotheses. This requires additional assumptions about appropriate probabilities of the hypotheses prior to consideration of the research evidence.