Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy

The past decade has seen the rise of evidence-based medicine, a movement that has focused attention on the importance of using clinical studies for empirical demonstration of the efficacy of medical interventions. Increasingly, physicians are being called on to assess such studies to help them make clinical decisions and understand the rationale behind recommended practices. This type of assessment requires an understanding of research methods that until recently was not expected of physicians. These research methods include statistical techniques used to assist in drawing conclusions. However, the methods of statistical inference in current use are not evidence-based and thus have contributed to a widespread misperception. The misperception is that absent any consideration of biological plausibility and prior evidence, statistical methods can provide a number that by itself reflects a probability of reaching erroneous conclusions. This belief has damaged the quality of scientific reasoning and discourse, primarily by making it difficult to understand how the strength of the evidence in a particular study can be related to and combined with the strength of other evidence (from other laboratory or clinical studies, scientific reasoning, or clinical experience). This results in many knowledge claims that do not stand the test of time (1, 2). A pair of articles in this issue examines this problem in some depth and proposes a partial solution. In this article, I explore the historical and logical foundations of the dominant school of medical statistics, sometimes referred to as frequentist statistics, which might be described as error-based. I explicate the logical fallacy at the heart of this system and the reason that it maintains such a tenacious hold on the minds of investigators, policymakers, and journal editors. In the second article (3), I present an evidence-based approach derived from Bayesian statistical methods, an alternative perspective that has been one of the most active areas of biostatistical development during the past 20 years. Bayesian methods have started to make inroads into medical journals; Annals, for example, has included a section on Bayesian data interpretation in its Information for Authors section since 1 July 1997. The perspective on Bayesian methods offered here will differ somewhat from that in previous presentations in other medical journals. It will focus not on the controversial use of these methods in measuring belief but rather on how they measure the weight of quantitative evidence. We will see how reporting an index called the Bayes factor (which in its simplest form is also called a likelihood ratio) instead of the P value can facilitate the integration of statistical summaries and biological knowledge and lead to a better understanding of the role of scientific judgment in the interpretation of medical research. An Example of the Problem A recent randomized, controlled trial of hydrocortisone treatment for the chronic fatigue syndrome showed a treatment effect that neared the threshold for statistical significance, P=0.06 (4). The discussion section began, hydrocortisone treatment was associated with an improvement in symptoms This is the first such study to demonstrate improvement with a drug treatment of [the chronic fatigue syndrome] (4). What is remarkable about this paper is how unremarkable it is. It is typical of many medical research reports in that a conclusion based on the findings is stated at the beginning of the discussion. Later in the discussion, such issues as biological mechanism, effect magnitude, and supporting studies are presented. But a conclusion is stated before the actual discussion, as though it is derived directly from the results, a mere linguistic transformation of P=0.06. This is a natural consequence of a statistical method that has almost eliminated our ability to distinguish between statistical results and scientific conclusions. We will see how this is a natural outgrowth of the P value fallacy. Philosophical Preliminaries To begin our exploration of the P value fallacy, we must consider the basic elements of reasoning. The process that we use to link underlying knowledge to the observed world is called inferential reasoning, of which there are two logical types: deductive inference and inductive inference. In deductive inference, we start with a given hypothesis (a statement about how nature works) and predict what we should see if that hypothesis were true. Deduction is objective in the sense that the predictions about what we will see are always true if the hypotheses are true. Its problem is that we cannot use it to expand our knowledge beyond what is in the hypotheses. Inductive inference goes in the reverse direction: On the basis of what we see, we evaluate what hypothesis is most tenable. The concept of evidence is inductive; it is a measure that reflects back from observations to an underlying truth. The advantage of inductive reasoning is that our conclusions about unobserved states of nature are broader than the observations on which they are based; that is, we use this reasoning to generate new hypotheses and to learn new things. Its drawback is that we cannot be sure that what we conclude about nature is actually true, a conundrum known as the problem of induction (5-7). From their clinical experience, physicians are acutely aware of the subtle but critical difference between these two perspectives. Enumerating the frequency of symptoms (observations) given the known presence of a disease (hypothesis) is a deductive process and can be done by a medical student with a good medical textbook (Figure 1, top). Much harder is the inductive art of differential diagnosis: specifying the likelihood of different diseases on the basis of a patient's signs, symptoms, and laboratory results. The deductions are more certain and objective but less useful than the inductions. Figure 1. The parallels between the processes of induction and deduction in medical inference ( top ) and statistical inference ( bottom ). The identical issue arises in statistics. Under the assumption that two treatments are the same (that is, the hypothesis of no difference in efficacy is true), it is easy to calculate deductively the frequency of all possible outcomes that we could observe in a study (Figure 1, bottom). But once we observe a particular outcome, as in the result of a clinical trial, it is not easy to answer the more important inductive question, How likely is it that the treatments are equivalent? In this century, philosophers have grappled with the problem of induction and have tried to solve or evade it in several ways. Karl Popper (8) proposed a philosophy of scientific practice that eliminated formal induction completely and used only the deductive elements of science: the prediction and falsification components. Rudolf Carnap tried an opposite strategyto make the inductive component as logically secure as the deductive part (9, 10). Both were unsuccessful in producing workable models for how science could be conducted, and their failures showed that there is no methodologic solution to the problem of fallible scientific knowledge. Determining which underlying truth is most likely on the basis of the data is a problem in inverse probability, or inductive inference, that was solved quantitatively more than 200 years ago by the Reverend Thomas Bayes. He withheld his discovery, now known as Bayes theorem; it was not divulged until 1762, 20 years after his death (11). Figure 2 shows Bayes theorem in words. Figure 2. Bayes theorem, in words. As a mathematical equation, Bayes theorem is not controversial; it serves as the foundation for analyzing games of chance and medical screening tests. However, as a model for how we should think scientifically, it is criticized because it requires assigning a prior probability to the truth of an idea, a number whose objective scientific meaning is unclear (7, 10, 12). It is speculated that this may be why Reverend Bayes chose the more dire of the publish or perish options. It is also the reason why this approach has been tarred with the subjective label and has not generally been used by medical researchers. Conventional (Frequentist) Statistical Inference Because of the subjectivity of the prior probabilities used in Bayes theorem, scientists in the 1920s and 1930s tried to develop alternative approaches to statistical inference that used only deductive probabilities, calculated with mathematical formulas that described (under certain assumptions) the frequency of all possible experimental outcomes if an experiment were repeated many times (10). Methods based on this frequentist view of probability included an index to measure the strength of evidence called the P value, proposed by R.A. Fisher in the 1920s (13), and a method for choosing between hypotheses, called a hypothesis test, developed in the early 1930s by the mathematical statisticians Jerzy Neyman and Egon Pearson (14). These two methods were incompatible but have become so intertwined that they are mistakenly regarded as part of a single, coherent approach to statistical inference (6, 15, 16). The P Value The P value is defined as the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed (Figure 3). Fisher proposed it as an informal index to be used as a measure of discrepancy between the data and the null hypothesis. It was not part of a formal inferential method. Fisher suggested that it be used as part of the fluid, non-quantifiable process of drawing conclusions from observations, a process that included combining the P value in some unspecified way with background information (17). Figure 3. The bell-shaped curve represents the probability of every possible outcome under the null hypothesis. P P It is worth noting one widely prevalent