A comparison of observational studies and randomized, controlled trials

BACKGROUND: For many years it has been claimed that observational studies find stronger treatment effects than randomized, controlled trials. We compared the results of observational studies with those of randomized, controlled trials. METHODS: We searched the Abridged Index Medicus and Cochrane data bases to identify observational studies reported between 1985 and 1998 that compared two or more treatments or interventions for the same condition. We then searched the Medline and Cochrane data bases to identify all the randomized, controlled trials and observational studies comparing the same treatments for these conditions. For each treatment, the magnitudes of the effects in the various observational studies were combined by the Mantel-Haenszel or weighted analysis-of-variance procedure and then compared with the combined magnitude of the effects in the randomized, controlled trials that evaluated the same treatment. RESULTS: There were 136 reports about 19 diverse treatments, such as calcium-channel-blocker therapy for coronary artery disease, appendectomy, and interventions for subfertility. In most cases, the estimates of the treatment effects from observational studies and randomized, controlled trials were similar. In only 2 of the 19 analyses of treatment effects did the combined magnitude of the effect in observational studies lie outside the 95 percent confidence interval for the combined magnitude in the randomized, controlled trials. CONCLUSIONS: We found little evidence that estimates of treatment effects in observational studies reported after 1984 are either consistently larger than or qualitatively different from those obtained in randomized, controlled trials.

[1]  Cass Principal Investigators and Their Associates.  Coronary artery surgery study (CASS): a randomized trial of coronary artery bypass surgery. Comparability of entry characteristics and survival in randomized patients and nonrandomized patients meeting randomization criteria. , 1984, Journal of the American College of Cardiology.

[2]  H. Genant,et al.  Postmenopausal bone loss is prevented by treatment with low-dosage estrogen with calcium. , 1987, Annals of internal medicine.

[3]  R N Hoover,et al.  Risk of leukemia after chemotherapy and radiation treatment for breast cancer. , 1992, The New England journal of medicine.

[4]  D P Byar,et al.  Using observational data from registries to compare treatments: the fallacy of omnimetrics. , 1984, Statistics in medicine.

[5]  T C Chalmers,et al.  Bias in treatment assignment in controlled clinical trials. , 1983, The New England journal of medicine.

[6]  A A Rimm,et al.  Mortality after coronary angioplasty and coronary artery bypass surgery (the national Medicare experience). , 1992, The American journal of cardiology.

[7]  I. Reid,et al.  Continuous combined oestrogen/progestin therapy is well tolerated and increases bone density at the hip and spine in post‐menopausal osteoporosis , 1994, Clinical endocrinology.

[8]  D. Kiel,et al.  The effect of postmenopausal estrogen therapy on bone density in elderly women. , 1993, The New England journal of medicine.

[9]  E. Emanuel.  Cost savings at the end of life. What do the data show? , 1996, JAMA.

[10]  A. Lane,et al.  Treatment of Postmenopausal Osteoporosis with Transdermal Estrogen , 1992, Annals of Internal Medicine.

[11]  C. Christiansen,et al.  Prevention of bone loss with alendronate in postmenopausal women under 60 years of age. , 1998 .

[12]  J. Aloia,et al.  Calcium Supplementation with and without Hormone Replacement Therapy To Prevent Postmenopausal Bone Loss , 1994, Annals of Internal Medicine.

[13]  G. Beck,et al.  Comparison of pneumatic retinopexy with alternative surgical techniques. , 1988, Ophthalmology.

[14]  V. L. Vallina,et al.  Laparoscopic versus conventional appendectomy. , 1993, Annals of surgery.

[15]  R. Boorse,et al.  Laparoscopic versus conventional appendectomy. , 1994, Journal of the American College of Surgeons.

[16]  D. Pohl,et al.  Laparoscopic versus open appendectomy: a metaanalysis. , 1998, Journal of the American College of Surgeons.

[17]  H. Mizunuma,et al.  Prevention of postmenopausal bone loss with minimal uterine bleeding using low dose continuous estrogen/progestin therapy: a 2-year prospective study. , 1997, Maturitas.

[18]  T C Chalmers,et al.  Randomized versus historical controls for clinical trials. , 1982, The American journal of medicine.

[19]  B. Psaty,et al.  Nifedipine. Dose-related increase in mortality in patients with coronary heart disease. , 1995, Circulation.

[20]  N. Mays,et al.  Results of one and two year follow-up in a clinical comparison of extracorporeal shock wave lithotripsy and percutaneous nephrolithotomy in the treatment of renal calculi. , 1992, Scandinavian journal of urology and nephrology.

[21]  B. Pruett,et al.  Laparoscopic appendectomy: have we found a better way? , 1994, Journal of the Mississippi State Medical Association.

[22]  P. Raskin,et al.  The effect of long-term near normal glycemic control on mild diabetic retinopathy. , 1985, Ophthalmology.

[23]  J. Heikkinen,et al.  HRT and exercise: effects on bone density, muscle strength and lipid metabolism. A placebo controlled 2-year prospective trial on two estrogen-progestin regimens in healthy postmenopausal women. , 1997, Maturitas.

[24]  R. Califf,et al.  Coronary revascularization in diabetic patients: a comparison of the randomized and observational components of the Aypass Angioplasty Revascularization Investigation (BARI). , 1999, Circulation.

[25]  M. Sugrue,et al.  Laparoscopic versus open appendicectomy: a prospective evaluation , 1993, The British journal of surgery.

[26]  W. Vaughn,et al.  The influence of antihypertensive drug therapy on renal transplant function and outcome. , 1988, Transplantation proceedings.

[27]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[28]  A D Oxman,et al.  The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials , 1998, BMJ.

[29]  S. Edge,et al.  Laparoscopic versus traditional appendectomy for suspected appendicitis. , 1993, American journal of surgery.

[30]  S. J. Smith,et al.  Effect of nifedipine on renal transplant rejection , 1993, The British journal of surgery.

[31]  Ladefoged Sd,et al.  Calcium channel blockers in kidney transplantation. , 1994 .

[32]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[33]  R B Haynes,et al.  The Cochrane Collaboration-Advances and Challenges in Improving Evidence-based Decision Making , 1998, Medical decision making : an international journal of the Society for Medical Decision Making.

[34]  T. Teasdale,et al.  A Comparison of Placement Outcomes of Geriatric Cohorts Receiving Care in a Geriatric Assessment Unit and on General Medicine Floors , 1983, Journal of the American Geriatrics Society.

[35]  C. Viscoli,et al.  Developing improved observational methods for evaluating therapeutic effectiveness. , 1990, The American journal of medicine.

[36]  M. Dören,et al.  Superior compliance and efficacy of continuous combined oral estrogen-progestogen replacement therapy in postmenopausal women. , 1995, American journal of obstetrics and gynecology.

[37]  A. Sagalowsky,et al.  IMPROVED OUTCOME OF CADAVERIC RENAL TRANSPLANTATION DUE TO CALCIUM CHANNEL BLOCKERS , 1991, Transplantation.

[38]  G H Guyatt,et al.  Users' guides to the medical literature. X. How to use an article reporting variations in the outcomes of health services. The Evidence-Based Medicine Working Group. , 1996, JAMA.

[39]  A. Oliveras,et al.  Usefulness of diltiazem in reducing the incidence of acute tubular necrosis in Euro-Collins-preserved cadaveric renal grafts. , 1991, Transplantation proceedings.

[40]  K Ottenbacher,et al.  Impact of random assignment on study outcome: an empirical examination. , 1992, Controlled clinical trials.

[41]  M A Hlatky,et al.  Comparison of predictions based on observational data with the results of randomized controlled clinical trials of coronary artery bypass surgery. , 1988, Journal of the American College of Cardiology.

[42]  P. Paricio,et al.  Laparoscopic versus open appendicectomy: A prospective assessment , 1994, The British journal of surgery.

[43]  P. Reichard,et al.  The effect of long-term intensified insulin treatment on the development of microvascular complications of diabetes mellitus. , 1993, The New England journal of medicine.

[44]  R. Lindsay,et al.  Postmenopausal osteoporosis: patient choices and outcomes. , 1995, Maturitas.

[45]  L. Rubenstein,et al.  Comprehensive geriatric assessment: a meta-analysis of controlled trials , 1993, The Lancet.

[46]  P. Tornambe,et al.  Pneumatic retinopexy. A multicenter randomized controlled clinical trial comparing pneumatic retinopexy with scleral buckling. The Retinal Detachment Study Group. , 1989, Ophthalmology.

[47]  H. Tiselius,et al.  Cost effectiveness of extracorporeal shock wave lithotripsy and percutaneous nephrolithotomy for medium-sized kidney stones. A randomised clinical trial. , 1992, Scandinavian journal of urology and nephrology.

[48]  J. Hanley,et al.  Trial of a Geriatric Consultation Team in an Acute Care Hospital , 1987, Journal of the American Geriatrics Society.

[49]  C Bain,et al.  Choosing between randomised and non-randomised studies: a systematic review. , 1998, Health technology assessment.

[50]  J. Salaman,et al.  Calcium-channel blockers and other factors influencing delayed function in renal allografts. , 1990, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[51]  S. Wimalawansa.  A four-year randomized controlled trial of hormone replacement and bisphosphonate, alone or in combination, in women with postmenopausal osteoporosis. , 1998, The American journal of medicine.

[52]  H. Rockette,et al.  Leukemia in breast cancer patients following adjuvant chemotherapy or postoperative radiation: the NSABP experience. , 1985, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[53]  U. Goldbourt,et al.  Calcium antagonists and mortality in patients with coronary artery disease: a cohort study of 11,575 patients. , 1996, Journal of the American College of Cardiology.

[54]  F. Mosteller,et al.  How study design affects outcomes in comparisons of therapy. I: Medical. , 1989, Statistics in medicine.

[55]  K. Nicolaides,et al.  Comparison of chorionic villus sampling and amniocentesis for fetal karyotyping at 10-13 weeks' gestation , 1994, The Lancet.

[56]  T C Chalmers,et al.  Evidence favoring the use of anticoagulants in the hospital phase of acute myocardial infarction. , 1977, The New England journal of medicine.

[57]  P. Raskin,et al.  Effect of glycemic control on microvascular complications in patients with type I diabetes mellitus. , 1986, The American journal of medicine.

[58]  R. W. Hughes,et al.  Treatment of benign esophageal stricture by Eder-Puestow or balloon dilators: a comparison between randomized and prospective nonrandomized trials. , 1992, Mayo Clinic proceedings.

[59]  R. Rhodes,et al.  Laparoscopic appendectomy: is it worth it? , 1994, The American surgeon.

[60]  A. Feinstein,et al.  Epidemiologic analyses of causation: the unlearned scientific lessons of randomized trials. , 1989, Journal of clinical epidemiology.

[61]  A. Genazzani,et al.  Effects of a new estrogen/progestin combination in the treatment of postmenopausal syndrome. , 1995, Maturitas.

[62]  D. Han,et al.  Comparison of pneumatic retinopexy and scleral buckling in the management of primary rhegmatogenous retinal detachment. Southern Wisconsin Pneumatic Retinopexy Study Group. , 1998, American journal of ophthalmology.

[63]  H. Genant,et al.  The comparative effect on bone density, endometrium, and lipids of continuous hormones as replacement therapy (CHART study). A randomized controlled trial. , 1996, JAMA.