Claims of Equivalence in Medical Research: Are They Supported by the Evidence?

Most clinical research activities are aimed at showing that one agent or method is better than another. An increasing number of reports, however, now conclude that the investigated entities are equivalent. Thus, a new drug or treatment may be deemed just as effective as a standard therapy while being, for example, less costly or easier to use (1). Equivalence can also be claimed for generic versions of innovator drugs (2) and for such diverse entities as medical protocols (3), surgical techniques (4), and medical devices (5). As physicians, insurers, and hospitals put increasing emphasis on practicing evidence-based medicine, claims of substantial treatment benefit have come under scrutiny. In contrast, claims of therapeutic equivalence may not be reviewed with the same quantitative rigor. This can lead to patient harm if clinically inferior treatments are erroneously deemed equivalent to a standard approach or if potentially superior therapies are discarded as merely equivalent. Despite the many scientific and statistical procedures used to confirm that a large difference is significant, less attention has been given to the logic and methods for establishing equivalence (6). In the context of hypothesis testing, equivalence exists only as a theoretical entityan infinitely large sample size would be needed to unequivocally establish no difference between compared groups. In practice, an observed difference can be compared to a specified value considered small (that is, not clinically important). Unfortunately, there are no established gold standard criteria for how to construct and support such an equivalence claim. Proposed approaches include using confidence intervals to exclude clinically meaningful differences (7) or applying variations of the analytic strategy of rejecting the null hypothesis (8). In this context, testing for equivalence involves rejecting an alternative hypothesis of a large difference between examined groups or entities. Yet, investigations that report clinical equivalence may not use this approach. Instead, after finding a negative result in conventional tests for statistical significance (for example, P >0.05), investigators may declare that the entities compared are equivalent. Our current study was done to determine whether published claims of equivalence are supported by the methods used and the results obtained. Methods Study Sample Using the National Library of Medicine Medical Subject Heading (MeSH) term therapeutic equivalency and the text word equivalence, we did a structured MEDLINE search for English-language original research reports published from 1992 through 1996 in which equivalence was claimed. The MeSH term searching process identifies papers in which the National Library of Medicine has identified a main point in the text. The separate text word searching process indicates papers in which the selected word is used in the title or abstract. We used both of these search strategies to obtain a representative, but not exhaustive, sample of published reports claiming equivalence. Reports identified with the specific MeSH term therapeutic equivalency are more likely to be methodologically sound, and papers defined by the text word equivalence are more likely to state the aim of the study than if they had been selected from a more general word, such as equivalent or equal. We reviewed the obtained citations on-line to select, for further evaluation, those that appeared likely to describe original research claiming equivalence. We included randomized clinical trials as well as pertinent observational studies (for example, reference 9) because although these types of research are conducted by using different methods, we believe that the quantitative claim of equivalence can be substantiated similarly. After reviewing the title and, if necessary, the abstract, we discarded citations to papers, such as reviews, meta-analyses, or commentaries (for example, references 10 and 11) that did not report original data. We also excluded reports (for example, references 12 and 13) of purely laboratory or other nonhuman research, as well as those intended solely to show pharmacokinetic bioequivalence, such as generic drug applications to the U.S. Food and Drug Administration. Finally, we excluded papers that reported equivalence for anything other than patients' outcomes, such as a comparison of two radiation therapy regimens in which dose equivalence referred only to standardized radiation fractions (14). For the remaining potentially eligible citations, abstracts were further reviewed according to the preceding criteria. Whenever the suitability of a paper was uncertain, the entire text was reviewed. All reviews were done in a structured fashion by one author using a database (Microsoft Access for Office 97, Microsoft, Inc., Redmond, Washington) created for this study. Although rules for inclusion and categorization were defined a priori, decisions were not straightforward for approximately 10% of reports. These difficult papers were reviewed by all three investigators for a consensus decision. Evaluation of Papers The entire text of each included paper was evaluated in a structured fashion for five prespecified attributes relating to the assessment and ultimate claim of equivalency. The attributes evaluated are listed below, along with the justification for their choice. 1. Statement of research aim. A stated research goal is needed to allow the investigators to choose the pertinent variables for study, boundaries for the magnitude of an equivalent result, and cogent analytic methods. 2. Magnitude of reported differences. Evaluating the clinical sensibility of an equivalence claim requires knowledge of exactly what is being called a negligibly small difference. We therefore tabulated the actual values of the quantities that constituted the difference between the investigated groups. Although these quantities ranged broadly in units of measurement and extent of variability, differences can be standardized as an effect size (15), which is usually calculated as the direct increment between groups divided by the standard deviation in the control group. For example, if drug X is successful in 60% 30% (SD) of patients and placebo is successful in 40% 20% (SD), then the effect size is (60% 40%)/20%=1.0. The effect size can thus be considered a unit-free ratio of signal to noise. 3. Choice of quantitative boundary. We determined whether the investigators had set an a priori quantitative boundary for what would constitute equivalence in the reports examined. Demarcating a maximum value of small, beyond which a difference could no longer be deemed equivalent, is needed for investigators and readers to appraise the numerical results in a clinical-scientific as well as a statistical manner. The boundary could be set from an absolute difference in means or medians, a proportionate difference in results, a ratio of two-group results, or an effect size. Although a single criterion does not exist for establishing what is large, proportionate differences of 20% or greater between clinical groups have been suggested as potentially important (16). In addition, an effect size less than 0.20 has often been considered trivially small, 0.50 has been considered moderate, and 0.80 has been considered large (17). We are not advocating general use of these arbitrary thresholds, and neither the incremental difference between compared groups, the proportional difference, nor the effect size was used as a gold standard criterion for what would constitute an acceptable study for equivalence. These thresholds can, however, serve to describe the magnitude of observed differences in our investigation, and this information is included for illustrative purposes. 4. Method of statistical (stochastic) testing. We next determined what, if any, testing was done to support the claim of equivalence, and we specifically checked whether the claim of equivalence was tested directly or was supported only by a failed test for superiority. In a direct test, the differences observed between groups or patients are compared against a specific equivalence boundary. A direct test of equivalency is intended to reject the alternative hypothesis that the true difference is larger than the boundary limit, whereas statistical tests for superiority are aimed at rejecting a null hypothesis of no difference. 5. Calculation of sample size. We expected that information about sample sizes, including advance calculations, would be reported. Such information would help, for example, to explain a paradox in which a large observed difference is deemed equivalent because the sample size was too small to achieve statistical significance. This problem of inadequate statistical power (which may cause studies to miss important observed differences) has been highlighted previously (18). Quality or Impact of Journals Although the quality of the journal might be considered when methodologic rigor is evaluated for published research, we did not try to define a high-quality medical journal. Instead, we determined which papers came from the 119 Abridged Index Medicus (AIM) journals that the National Library of Medicine regards as selected biomedical journal literature of immediate interest to the practicing physician (19). The National Library of Medicine states that journals are included in AIM according to the quality of the journal, usefulness of journal content for the professional, and the need for providing coverage in the fields of clinical medicine (19). The list includes Annals of Internal Medicine, BMJ, JAMA, The Lancet, The New England Journal of Medicine, and many leading subspecialty journals but omits such nonclinical journals as Science, Nature, and Cell. Role of the Funding Source This study was funded by the Robert Wood Johnson Foundation through its Clinical Scholars Program. The funding source had no role in the collection, analysis, or interp