No short-cut in assessing trial quality: a case study

BackgroundAssessing the quality of included trials is a central part of a systematic review. Many check-list type of instruments for doing this exist. Using a trial of antibiotic treatment for acute otitis media, Burke et al., BMJ, 1991, as the case study, this paper illustrates some limitations of the check-list approach to trial quality assessment.ResultsThe general verdict from the check list type evaluations in nine relevant systematic reviews was that Burke et al. (1991) is a good quality trial. All relevant meta-analyses extensively used its data to formulate therapeutic evidence. My comprehensive evaluation, on the other hand, brought to the surface a series of serious problems in the design, conduct, analysis and report of this trial that were missed by the earlier evaluations.ConclusionA check-list or instrument based approach, if used as a short-cut, may at times rate deeply flawed trials as good quality trials. Check lists are crucial but they need to be augmented with an in-depth review, and where possible, a scrutiny of the protocol, trial records, and original data. The extent and severity of the problems I uncovered for this particular trial warrant an independent audit before it is included in a systematic review.

[1]  S. Coughlin,et al.  Systematic Reviews in Health Care: A Practical Guide , 2002 .

[2]  A. Hoes,et al.  Antibiotics for acute otitis media: a meta-analysis with individual patient data , 2006, The Lancet.

[3]  Douglas G Altman,et al.  Systematic reviews in health care: Assessing the quality of controlled clinical trials. , 2001, BMJ.

[4]  C. Farquhar,et al.  False Hope: Bone Marrow Transplantation for Breast Cancer , 2007 .

[5]  A. Laupacis,et al.  Systematic Reviews: Time to Address Clinical and Policy Relevance As Well As Methodological Rigor , 2007, Annals of Internal Medicine.

[6]  I. Chalmers,et al.  A cohort study of summary reports of controlled trials. , 1990, JAMA.

[7]  Douglas G Altman,et al.  Epidemiology and reporting of randomised trials published in PubMed journals , 2005, The Lancet.

[8]  P. Gøtzsche,et al.  Flaws in design, analysis and interpretation of Pfizer's antifungal trials of voriconazole and uncritical subsequent quotations , 2006, Trials.

[9]  J. Hilden,et al.  Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. , 2007, International journal of epidemiology.

[10]  W. Blackwelder,et al.  Treatment of acute otitis media. , 1982, JAMA.

[11]  C. Marchant,et al.  Antibacterial therapy for acute otitis media: a critical analysis. , 1982, Reviews of infectious diseases.

[12]  A. Vail,et al.  Common statistical errors in the design and analysis of subfertility trials. , 2003, Human reproduction.

[13]  A. Hrõbjartsson,et al.  Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding. , 2007, International journal of epidemiology.

[14]  R. Feigin.  Otitis media: closing the information gap. , 1982, The New England journal of medicine.

[15]  J. Robinson,et al.  Management of acute otitis media. , 2009, Paediatrics & child health.

[16]  G. Hordijk,et al.  A review of clinical trials regarding treatment of acute otitis media. , 1992, Clinical otolaryngology and allied sciences.

[17]  S. Berman.  Otitis media in children. , 1995, The New England journal of medicine.

[18]  F. V. Buchem,et al.  THERAPY OF ACUTE OTITIS MEDIA: MYRINGOTOMY, ANTIBIOTICS, OR NEITHER? A Double-blind Study in Children , 1981, The Lancet.

[19]  Many Reviews Are Systematic but Some Are More Transparent and Completely Reported than Others , 2007, PLoS medicine.

[20]  D. Cook,et al.  Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. , 1999, Health technology assessment.

[21]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[22]  K. Schulz.  Assessing allocation concealment and blinding in randomised controlled trials: why bother? , 2000, Evidence-based nursing.

[23]  J. Marshall.  Are antibiotics indicated as initial treatment for children with acute otitis media? A meta-analysis , 1997 .

[24]  N. Persaud,et al.  External validity: the neglected dimension in evidence ranking. , 2006, Journal of evaluation in clinical practice.

[25]  David R. Jones,et al.  Methods for the analysis of quality-of-life and survival data in health technology assessment. , 1999, Health technology assessment.

[26]  Ethan M Balk,et al.  Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. , 2002, JAMA.

[27]  D. Altman.  Poor-quality medical research: what can journals do? , 2002, JAMA.

[28]  E. Cantekin.  Aggressive and Ineffective Therapy for Otitis media , 1998, Oto-Rhino-Laryngologia Nova.

[29]  Kent Johnson,et al.  The power of the protocol , 2002, The Lancet.

[30]  F. V. Buchem,et al.  THERAPY OF ACUTE OTITIS MEDIA , 1981, The Lancet.

[31]  D. Altman,et al.  Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research , 2004, Canadian Medical Association Journal.

[32]  P. Rothwell,et al.  Factors That Can Affect the External Validity of Randomised Controlled Trials , 2006, PLoS clinical trials.

[33]  C. Adams,et al.  Content and quality of 2000 controlled trials in schizophrenia over 50 years , 1998, BMJ.

[34]  T. Lehnert,et al.  Acute otitis media in children. Role of antibiotic therapy. , 1993, Canadian family physician Medecin de famille canadien.

[35]  D. Rennie,et al.  Measuring the quality of trials: the quality of quality scales. , 1999, JAMA.

[36]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[37]  S. Greenland.  Quality Scores Are Useless and Potentially Misleading: Reply to “Re: A Critical Look at Some Popular Analytic Methods” , 1994 .

[38]  David Moher,et al.  Epidemiology and Reporting Characteristics of Systematic Reviews , 2007, PLoS medicine.

[39]  A. Jadad,et al.  The importance of quality of primary studies in producing unbiased systematic reviews. , 1996, Archives of internal medicine.

[40]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[41]  M. Bracken,et al.  Clinically useful measures of effect in binary analyses of randomized trials. , 1994, Journal of clinical epidemiology.

[42]  B. Djulbegovic,et al.  Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group , 2004, BMJ : British Medical Journal.

[43]  L Culpepper,et al.  Antimicrobials for acute otitis media? a review from the international primary care network , 1997, BMJ.

[44]  A. Hoes,et al.  Primary care based randomised, double blind trial of amoxicillin versus placebo for acute otitis media in children aged under 2 years , 2000, BMJ : British Medical Journal.

[45]  R. Rosenfeld,et al.  Evidence-Based Otitis Media , 2003 .

[46]  J. Emmerich,et al.  OTITIS MEDIA: MICROBIOLOGY AND EVALUATION OF THERAPY * , 1967, Annals of the New York Academy of Sciences.

[47]  P. Burke,et al.  Acute red ear in children: controlled trial of non-antibiotic treatment in general practice. , 1991, BMJ.

[48]  K. Schulz.  Assessing allocation concealment and blinding in randomised controlled trials: why bother? , 2000, Equine veterinary journal.

[49]  Graham A. Colditz,et al.  Systematic Reviews in Health Care: Frontmatter , 2001 .

[50]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[51]  J. Froom,et al.  Routine antimicrobial treatment of acute otitis media: is it necessary? , 1997, JAMA.

[52]  Emil H Schemitsch,et al.  Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. , 2004, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[53]  J. Sissons.  Skin lesions, angioedema, and hypocomplementemia , 1974 .

[54]  R. Rosenfeld,et al.  Clinical efficacy of antimicrobial drugs for acute otitis media: metaanalysis of 5400 children from thirty-three randomized trials. , 1994, The Journal of pediatrics.

[55]  M. Cho,et al.  Instruments for assessing the quality of drug studies published in the medical literature. , 1994, JAMA.

[56]  T C Chalmers,et al.  Bias in treatment assignment in controlled clinical trials. , 1983, The New England journal of medicine.

[57]  H. Krumholz,et al.  What have we learnt from Vioxx? , 2007, BMJ : British Medical Journal.

[58]  G Lewis,et al.  Putting trials on trial--the costs and consequences of small trials in depression: a systematic review of methodology. , 1997, Journal of epidemiology and community health.