Randomized, controlled trials, observational studies, and the hierarchy of research designs.

BACKGROUND In the hierarchy of research designs, the results of randomized, controlled trials are considered to be evidence of the highest grade, whereas observational studies are viewed as having less validity because they reportedly overestimate treatment effects. We used published meta-analyses to identify randomized clinical trials and observational studies that examined the same clinical topics. We then compared the results of the original reports according to the type of research design. METHODS A search of the Medline data base for articles published in five major medical journals from 1991 to 1995 identified meta-analyses of randomized, controlled trials and meta-analyses of either cohort or case-control studies that assessed the same intervention. For each of five topics, summary estimates and 95 percent confidence intervals were calculated on the basis of data from the individual randomized, controlled trials and the individual observational studies. RESULTS For the five clinical topics and 99 reports evaluated, the average results of the observational studies were remarkably similar to those of the randomized, controlled trials. For example, analysis of 13 randomized, controlled trials of the effectiveness of bacille Calmette-Guérin vaccine in preventing active tuberculosis yielded a relative risk of 0.49 (95 percent confidence interval, 0.34 to 0.70) among vaccinated patients, as compared with an odds ratio of 0.50 (95 percent confidence interval, 0.39 to 0.65) from 10 case-control studies. In addition, the range of the point estimates for the effect of vaccination was wider for the randomized, controlled trials (0.20 to 1.56) than for the observational studies (0.17 to 0.84). CONCLUSIONS The results of well-designed observational studies (with either a cohort or a case-control design) do not systematically overestimate the magnitude of the effects of treatment as compared with those in randomized, controlled trials on the same topic.

[1]  J. Ware,et al.  Randomized clinical trials. Perspectives on some recent ideas. , 1976, The New England journal of medicine.

[2]  A. Feinstein,et al.  Methodologic standards and contradictory results in case-control research , 1979 .

[3]  T C Chalmers,et al.  Randomized versus historical controls for clinical trials. , 1982, The American journal of medicine.

[4]  T C Chalmers,et al.  Bias in treatment assignment in controlled clinical trials. , 1983, The New England journal of medicine.

[5]  T C Chalmers,et al.  Sensitivity and Specificity of Clinical Trials: Randomized v Historical Controls , 1983 .

[6]  A. Feinstein.  Current problems and future challenges in randomized clinical trials. , 1984, Circulation.

[7]  N. Laird,et al.  Meta-analysis in clinical trials. , 1986, Controlled clinical trials.

[8]  R. Horwitz,et al.  Complexity and contradiction in clinical trial research. , 1987, The American journal of medicine.

[9]  R. Collins,et al.  Blood pressure, stroke, and coronary heart disease Part 2, short-term reductions in blood pressure: overview of randomised drug trials in their epidemiological context , 1990, The Lancet.

[10]  C. Viscoli,et al.  Developing improved observational methods for evaluating therapeutic effectiveness. , 1990, The American journal of medicine.

[11]  R. Collins,et al.  Blood pressure, stroke, and coronary heart disease Part 1, prolonged differences in blood pressure: prospective observational studies corrected for the regression dilution bias , 1990, The Lancet.

[12]  A. Detsky,et al.  Evidence-based medicine. A new approach to teaching the practice of medicine. , 1992, JAMA.

[13]  S. Yusuf,et al.  Report of the Conference on Low Blood Cholesterol: Mortality Associations , 1992, Circulation.

[14]  J. Fleiss.  Review papers : The statistical basis of meta-analysis , 1993 .

[15]  M. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, American Psychologist.

[16]  F. Mosteller,et al.  Efficacy of BCG Vaccine in the Prevention of Tuberculosis: Meta-analysis of the Published Literature , 1994 .

[17]  B. Psaty,et al.  The Association between Cholesterol and Death from Injury , 1994, Annals of Internal Medicine.

[18]  S. Rubin,et al.  Efficacy of screening mammography. A meta-analysis. , 1995, JAMA.

[19]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[20]  G. Guyatt,et al.  Users' Guides to the Medical Literature: IX. A Method for Grading Health Care Recommendations , 1995 .

[21]  R J Cook,et al.  Users' guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. , 1995, JAMA.

[22]  U. P. S. T. Force.  Guide to Clinical Preventive Services: Report of the U S Preventive Services Task Force , 1996 .

[23]  G. Grégoire,et al.  Discrepancies between meta-analyses and subsequent large randomized, controlled trials. , 1997, The New England journal of medicine.

[24]  J P Kassirer,et al.  Alternative medicine--the risks of untested and unregulated remedies. , 1998, The New England journal of medicine.

[25]  K. D. Horn.  Evolving strategies in the treatment of sepsis and systemic inflammatory response syndrome (SIRS). , 1998, QJM : monthly journal of the Association of Physicians.

[26]  A D Oxman,et al.  The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials , 1998, BMJ.

[27]  K. Demissie,et al.  Empirical comparison of the results of randomized controlled trials and case-control studies in evaluating the effectiveness of screening mammography. , 1998, Journal of clinical epidemiology.

[28]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[29]  U. Abel,et al.  The role of randomization in clinical studies: myths and beliefs. , 1999, Journal of clinical epidemiology.

[30]  C Bain,et al.  Interpreting the evidence: choosing between randomised and non-randomised studies , 1999, BMJ.

[31]  J. A. Kruse,et al.  E5 murine monoclonal antiendotoxin antibody in gram-negative sepsis: a randomized controlled trial. E5 Study Investigators. , 2000, JAMA.