A newly developed tool for classifying study designs in systematic reviews of interventions and exposures showed substantial reliability and validity.

OBJECTIVE To develop a study Design Algorithm for Medical Literature on Intervention (DAMI) and test its interrater reliability, construct validity, and ease of use. STUDY DESIGN AND SETTING We developed and then revised the DAMI to include detailed instructions. To test the DAMI's reliability, we used a purposive sample of 134 primary, mainly nonrandomized studies. We then compared the study designs as classified by the original authors and through the DAMI. Unweighted kappa statistics were computed to test interrater reliability and construct validity based on the level of agreement between the original and DAMI classifications. Assessment time was also recorded to evaluate ease of use. RESULTS The DAMI includes 13 study designs, including experimental and observational studies of interventions and exposure. Both the interrater reliability (unweighted kappa = 0.67; 95% CI [0.64-0.75]) and construct validity (unweighted kappa = 0.63, 95% CI [0.52-0.67]) were substantial. Mean classification time using the DAMI was 4.08 ± 2.44 minutes (range, 0.51-10.92). CONCLUSIONS The DAMI showed substantial interrater reliability and construct validity. Furthermore, given its ease of use, it could be used to accurately classify medical literature for systematic reviews of interventions although minimizing disagreement between authors of such reviews.

[1]  Carolyn Nickson,et al.  Mammographic Screening and Breast Cancer Mortality: A Case–Control Study and Meta-analysis , 2012, Cancer Epidemiology, Biomarkers & Prevention.

[2]  Thomas Rotter,et al.  Clinical pathways: effects on professional practice, patient outcomes, length of stay and hospital costs. , 2010, The Cochrane database of systematic reviews.

[3]  J. Sim,et al.  The kappa statistic in reliability studies: use, interpretation, and sample size requirements. , 2005, Physical therapy.

[4]  J. R. Landis,et al.  An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. , 1977, Biometrics.

[5]  R. Fletcher,et al.  Clinical Epidemiology: The Essentials , 1982 .

[6]  Paul G Shekelle,et al.  Assessment of a method to detect signals for updating systematic reviews , 2014, Systematic Reviews.

[7]  T. Jefferson,et al.  Vaccines for preventing influenza in healthy children. , 2012, The Cochrane database of systematic reviews.

[8]  S. Cummings,et al.  Designing Clinical Research: An Epidemiologic Approach , 1988 .

[9]  Scott D Ramsey,et al.  Observational study designs for comparative effectiveness research: an alternative approach to close evidence gaps in head-and-neck cancer. , 2014, International journal of radiation oncology, biology, physics.

[10]  Jeffrey J. Walline,et al.  Designing Clinical Research: an Epidemiologic Approach, 2nd Ed. , 2001 .

[11]  D. Altman,et al.  Distinguishing Case Series From Cohort Studies , 2012, Annals of Internal Medicine.

[12]  Tom Jefferson,et al.  Vaccines for preventing influenza in the elderly. , 2010, The Cochrane database of systematic reviews.

[13]  Suzanne Bakken,et al.  Development and Evaluation of a Study Design Typology for Human Research , 2009, AMIA.

[14]  J. Critchley,et al.  Smoking cessation for the secondary prevention of coronary heart disease. , 2012, The Cochrane database of systematic reviews.

[15]  S. Boufous,et al.  Helmets for preventing injury in motorcycle riders. , 2008, The Cochrane database of systematic reviews.

[16]  D. Cook,et al.  Effect of smoking cessation on mortality after myocardial infarction: meta-analysis of cohort studies. , 2000, Archives of internal medicine.

[17]  R S Chrzanowski,et al.  Study design for technology assessment: critical issues. , 1988, Health policy.

[18]  P. Briss,et al.  Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services. , 2000, American journal of preventive medicine.

[19]  Lisa Hartling,et al.  Observational evidence and strength of evidence domains: case examples , 2014, Systematic Reviews.

[20]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration , 2013 .

[21]  Stephanie M. Chang,et al.  The Agency for Healthcare Research and Quality (AHRQ) effective health care (EHC) program methods guide for comparative effectiveness reviews: keeping up-to-date in a rapidly evolving field. , 2011, Journal of clinical epidemiology.

[22]  Tom Jefferson,et al.  Physical interventions to interrupt or reduce the spread of respiratory viruses: systematic review , 2008, BMJ : British Medical Journal.

[23]  A. Viera,et al.  Understanding interobserver agreement: the kappa statistic. , 2005, Family medicine.

[24]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.