Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: A meta-review.

At present, there exists no widely agreed upon set of study-design selection criteria for systematic reviews of health systems research, except for those proposed by the Cochrane Collaboration's Effective Practice and Organisation of Care (EPOC) review group (which comprises randomized controlled trials, controlled clinical trials, controlled before-after studies, and interrupted time series). We conducted a meta-review of the study-design selection criteria used in systematic reviews available in the McMaster University's Health Systems Evidence or the EPOC database. Of 414 systematic reviews, 13% did not indicate any study-design selection criteria. Of the 359 studies that described such criteria, 50% limited their synthesis to controlled trials and 68% to some or all of the designs defined by the EPOC criteria. Seven out of eight reviews identified at least one controlled trial that was relevant for the review topic. Seven percent of the reviews included either no or only one relevant primary study. Our meta-review reveals reviewers' preferences for restricting synthesis to controlled experiments or study designs that comply with the EPOC criteria. We discuss the advantages and disadvantages of the current practices regarding study-design selection in systematic reviews of health systems research as well as alternative approaches.

[1]  J. Newhouse,et al.  Econometrics in outcomes research: the use of instrumental variables. , 1998, Annual review of public health.

[2]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[3]  Andrea Milne,et al.  Diabetes education for children with type 1 diabetes mellitus and their families. , 2008, Evidence report/technology assessment.

[4]  M. Schützwohl,et al.  Involuntary vs. voluntary hospital admission. A systematic literature review on outcome diversity. , 2008, European archives of psychiatry and clinical neuroscience.

[5]  K. Humphreys,et al.  Use of exclusion criteria in selecting research subjects and its effect on the generalizability of alcohol treatment outcome studies. , 2000, The American journal of psychiatry.

[6]  Fiona Stewart,et al.  The Cochrane collaboration's tool for assessing risk of bias , 2013 .

[7]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[8]  Y. van der Graaf,et al.  The outcomes of ovarian cancer treatment are better when provided by gynecologic oncologists and in specialized hospitals: a systematic review. , 2007, Gynecologic oncology.

[9]  Shobha Phansalkar,et al.  The state of the evidence for computerized provider order entry: A systematic review and analysis of the quality of the literature , 2009, Int. J. Medical Informatics.

[10]  Sara Bennett,et al.  The Use of Research Evidence in Two International Organizations' Recommendations About Health Systems , 2009, Healthcare policy = Politiques de sante.

[11]  A. Farmer,et al.  Designing and evaluating complex interventions to improve health care , 2007, BMJ : British Medical Journal.

[12]  Andrew D Oxman,et al.  SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking? , 2009, Health Research Policy and Systems.

[13]  Rainer Sauerborn,et al.  One hundred and eighteen years of the German health insurance system: are there any lessons for middle- and low-income countries? , 2002, Social science & medicine.

[14]  Andrew D Oxman,et al.  SUPPORT Tools for evidence-informed health Policymaking (STP) , 2009, Health research policy and systems.

[15]  John N. Lavis,et al.  How Can We Support the Use of Systematic Reviews in Policymaking? , 2009, PLoS medicine.

[16]  J. Popay,et al.  Informing policy making and management in healthcare: the place for synthesis. , 2006, Healthcare policy = Politiques de sante.

[17]  Nicola J Cooper,et al.  Evidence synthesis as the key to more coherent and efficient research , 2009, BMC medical research methodology.

[18]  Wiebe R. Pestman,et al.  Instrumental Variables: Application and Limitations , 2006, Epidemiology.

[19]  Alexander C. Wagenaar,et al.  The Value of Interrupted Time-Series Experiments for Community Intervention Research , 2000, Prevention Science.

[20]  K. Humphreys,et al.  Ineligibility and refusal to participate in randomised trials of treatments for drug dependence. , 2009, Drug and alcohol review.

[21]  J. Macinko,et al.  The Brazilian health system: history, advances, and challenges , 2011, The Lancet.

[22]  Sara Bennett,et al.  Effects of policy options for human resources for health: an analysis of systematic reviews , 2008, The Lancet.

[23]  M. Petticrew,et al.  Alternatives to randomisation in the evaluation of public health interventions: design challenges and solutions , 2009, Journal of Epidemiology & Community Health.

[24]  J. Popay,et al.  Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field , 2005, Journal of health services research & policy.

[25]  James J. Heckman,et al.  Assessing the Case for Social Experiments , 1995 .

[26]  J Thomas,et al.  Randomised controlled trials for policy interventions: a review of reviews and meta-regression. , 2010, Health technology assessment.

[27]  S. Bennett,et al.  Priority setting and health policy and systems research , 2009, Health research policy and systems.

[28]  A. Deaton.  Instruments of Development: Randomization in the Tropics, and the Search for the Elusive Keys to Economic Development , 2009 .

[29]  J. M. Davies,et al.  Qualitative Research in Health Care , 1996, Journal of the Royal College of Physicians of London.

[30]  C. McKevitt,et al.  Preventing stroke: a narrative review of community interventions for improving hypertension control in black adults. , 2007, Health & social care in the community.

[31]  B. Stuart,et al.  Application of regression-discontinuity analysis in pharmaceutical health services research. , 2006, Health services research.

[32]  S. Bose,et al.  The effectiveness of contracting-out primary health care services in developing countries: a review of the evidence. , 2007, Health policy and planning.

[33]  Stephan D. Fihn,et al.  A Review of Interventions and System Changes to Improve Time to Reperfusion for ST-Segment Elevation Myocardial Infarction , 2008, Journal of General Internal Medicine.

[34]  E. Foster,et al.  Propensity Score Matching: An Illustrative Analysis of Dose Response , 2003, Medical care.

[35]  Kathryn H. Bowles,et al.  Applying Research Evidence to Optimize Telehomecare , 2007, The Journal of cardiovascular nursing.

[36]  G. Guyatt,et al.  GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). , 2011, Journal of clinical epidemiology.

[37]  Z. Bhutta,et al.  Impact of packaged interventions on neonatal health: a review of the evidence. , 2007, Health policy and planning.

[38]  M. Roemer.  National health systems of the world , 1991 .

[39]  D. Howard,et al.  The impact of waiting time on liver transplant outcomes. , 2000, Health services research.

[40]  David Card,et al.  The Impact of Nearly Universal Insurance Coverage on Health Care Utilization: Evidence from Medicare. , 2008, The American economic review.

[41]  David Ogilvie,et al.  Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go? , 2005, Journal of Epidemiology and Community Health.

[42]  J. Angrist,et al.  Journal of Economic Perspectives—Volume 15, Number 4—Fall 2001—Pages 69–85 Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments , 2022 .

[43]  Laurie E Felland,et al.  Qualitative Methods: A Crucial Tool for Understanding Changes in Health Systems and Health Care Delivery , 2011, Medical care research and review : MCRR.

[44]  G. Guyatt,et al.  The Science of Reviewing Research a , 1993, Annals of the New York Academy of Sciences.

[45]  Sean D Sullivan,et al.  Methods to assess intended effects of drug treatment in observational studies are reviewed. , 2004, Journal of clinical epidemiology.

[46]  Systematic reviews in public health: old chestnuts and new challenges. , 2009, Bulletin of the World Health Organization.

[47]  Partha Deb,et al.  When and how to use instrumental variables in palliative care research. , 2009, Journal of palliative medicine.

[48]  G. Antes.  The new CONSORT statement , 2010, BMJ : British Medical Journal.

[49]  C. Pope,et al.  Qualitative Research: Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research , 1995 .