Assessing the quality of randomized controlled trials: Quality of design is not the only relevant variable

Few would disagree with the proposition that ethical physicians should base their treatment of patients on the absolute truth concerning the efficacy, side effects, and other consequences of the available therapeutic options. However, absolute truth is hard to come by and nowhere more difficult to achieve than in the arena of clinical research. Here, a variety of biases have been shown repeatedly to interpose themselves between what is true, what is observed, and what is reported. Their nature spans a broad spectrum from the most egregious, in which personal gain colors judgment, to highly subtle issues that allow entirely innocent and unrecognized but critical biases to alter the conduct and interpretation of therapeutic trials. The scientific method as an approach to specific aspects of absolute truth has evolved over the past three centuries, most notably within the past 75 years. Its growing application to clinical medicine and the increasing reliance on clinical trials as a means for getting to the truth, at least with regard to therapeutics, is largely a post–World War II phenomenon. The development of government-sponsored cooperative study groups to carry out large, multicenter clinical trials in cancer, cardiovascular disease, and other fields was well established by the 1960s, stimulating—and stimulated by—major advances in what we now consider the discipline of biostatistics. The importance of avoiding bias in clinical studies within the discipline of hepatology burst on our scene in the 1970s. More than any other individual, Dr. Thomas C. Chalmers, sequentially a president of the American Association for the Study of Liver Diseases, director of the Clinical Center of the National Institutes of Health, and president of the Mount Sinai Medical Center and dean of its School of Medicine, was responsible for this enlightenment. In part from his own earlier experience as a clinical investigator, he recognized the importance of randomization, blinding, and predetermination of appropriate sample sizes as essential to the design of clinical trials that avoided both type I and type II statistical errors in formulating their conclusions.1,2 He also recognized that well done ‘‘negative’’ trials were an important source of information and that the editorial prejudice against their publication introduced bias into the literature as a whole. Deeply committed to a biostatistically based approach, he sought to illustrate its power in every conceivable arena. To show how subtle biases may influence outcomes, he conducted among NIH employees an informal, four-way randomized but nonblinded trial of vitamin C for treating the common cold. The best results were in the group that believed it was getting vitamin C but was, in reality, taking a placebo. Similarly, at his dinner table it was obligatory for guests to rank the quality of the wines served, some bottles being unmarked, others bearing a label but not necessarily the original one. As anticipated, palatal judgment was often influenced by the label. With increasing reliance on the randomized controlled clinical trial as the principal guide to therapeutic truth, it has become apparent that trials are subject to bias unless conducted in very specific ways. Thus, when the day of the week determines the treatment arm to which a patient is assigned, the distribution may not be truly ‘‘random,’’ because investigators may manipulate the system by delaying patient entry. The importance of blinding of the patient so as to avoid the well-known placebo effect is clearly illustrated by Dr. Chalmers’ vitamin C trial, but failure to blind the investigator can lead to equally important biases. As these issues and their potential to influence the results of randomized trials were recognized, it became increasingly clear that, in a truly Orwellian sense, although almost any structured trial was better than purely anecdotal experience, some trials were ‘‘more better’’ than others. The issue took on added importance with the growth of meta-analysis as a way to increase statistical power by pooling numerous small trials. The inclusion of poorly designed trials with potentially biased data could alter the conclusions of the entire meta-analysis.3,4 Attempts to evaluate the methodological rigor of published clinical trials go back nearly four decades.5 Since that time, many systems have evolved for grading trials based on the presence or absence of measures to prevent bias. Grading could take into account a single component of trial design considered to be especially important (e.g. the randomization procedure); alternatively, it could involve an extensive checklist, such as one developed by the Chalmers group6; or it could assess a smaller number of components, essentially a short checklist, and arrive at a numerical score.7 The article in this issue, ‘‘Randomized Clinical Trials in HEPATOLOGY: Predictors of Quality’’ by Kjaergard, Nikolova, and Gluud,8 uses a short list and numerical score. In it, the authors grade all 235 randomized clinical trials (RCTs) published in HEPATOLOGY from its initial issue in 1981 through August, 1998. The assessment comprises essentially three items: the way in which patients were allocated to study groups (0-2 points); double blinding (0-2 points); and an accounting of patients who dropped out or withdrew (0-1 point). The authors examine also two other design components: whether there was adequate concealment of the Abbreviations: RCT, randomized clinical trial. From the 1Department of Medicine (Division of Liver Disease) and 2Thomas C. Chalmers Clinical Trials Unit, Mount Sinai School of Medicine, New York, NY. Received September 13, 1999; accepted September 20, 1999. Address reprint requests to: Paul D. Berk, M.D., Mount Sinai School of Medicine (Box 1633), 1 Gustave L. Levy Place, New York, NY 10029. E-mail: paul_berk@smtplink. mssm.edu; fax: 212-348-3517. Copyright r 1999 by the American Association for the Study of Liver Diseases. 0270-9139/99/3005-0032$3.00/0

[1]  C. Gluud,et al.  Randomized clinical trials in Hepatology: Predictors of quality , 1999, Hepatology.

[2]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[3]  C. Gluud Evidence based medicine in LIVER. , 1999, Liver.

[4]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[5]  C. Gluud,et al.  Quality assessment of reports on clinical trials in the Journal of Hepatology. , 1998, Journal of hepatology.

[6]  L. Opie Conflict of interest in the debate over calcium-channel antagonists. , 1998, The New England journal of medicine.

[7]  I. Olkin,et al.  Improving the quality of reporting of randomized controlled trials. The CONSORT statement. , 1996, JAMA.

[8]  H P Friedman,et al.  Meta‐analysis: An introduction and point of view , 1996, Hepatology.

[9]  A. Jadad,et al.  The importance of quality of primary studies in producing unbiased systematic reviews. , 1996, Archives of internal medicine.

[10]  A R Jadad,et al.  Assessing the Quality of Randomized Controlled Trials: Current Issues and Future Directions , 1996, International Journal of Technology Assessment in Health Care.

[11]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[12]  T C Chalmers,et al.  Bias in treatment assignment in controlled clinical trials. , 1983, The New England journal of medicine.

[13]  T C Chalmers,et al.  A method for assessing the quality of a randomized control trial. , 1981, Controlled clinical trials.

[14]  T C Chalmers,et al.  The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial. Survey of 71 "negative" trials. , 1978, The New England journal of medicine.

[15]  R. Badgley,et al.  An assessment of research methods reported in 103 scientific articles from two Canadian medical journals. , 1961, Canadian Medical Association journal.