The nature of truth: Simpson's Paradox and the limits of statistical data.

‘Give me a fruitful error any time, full of seeds, bursting with its own corrections. You can keep your sterile truth for yourself.’ Vilfredo Pareto We usually think in terms of true and false, and often believe that we know which is which. Nonetheless, sometimes information which appears to be true is in fact false. Although we try to base our medical knowledge on objective evidence—research and statistics—rather than our personal opinions, Simpson's Paradox reminds us of the limitations of statistical evidence. In this phenomenon, an apparent paradox arises because aggregated data can support a conclusion which is opposite from that suggested by the same data before aggregation. One would generally conclude from the data in Table 1 that treatment A is the treatment of choice for the condition studied (given that side‐effects are equal). Suppose, however, that these patients consisted of two subgroups: those with a high serum level of substance X, and those with a low level. Table 2 shows the data for the patients with high serum X. For this subgroup of patients, treatment B seems to be better than treatment A. Since A is the preferable treatment in the group as a whole, one might intuitively expect the other patients to be better off with treatment A. But this is not the case (Table 3). Even in patients with low serum X, treatment B is still better (although fewer of these patients benefit from either treatment). Thus, if the patient's serum X level is unknown, treatment A seems to be better, but if serum X is known, treatment B is preferable (and one can better predict the response rate of a patient). This phenomenon is a result of the aggregation of two (or more) subgroups.1 The numbers of the example are kept simple to demonstrate …