An Overview of Research and Evaluation Designs for Dissemination and Implementation.

The wide variety of dissemination and implementation designs now being used to evaluate and improve health systems and outcomes warrants review of the scope, features, and limitations of these designs. This article is one product of a design workgroup that was formed in 2013 by the National Institutes of Health to address dissemination and implementation research, and whose members represented diverse methodologic backgrounds, content focus areas, and health sectors. These experts integrated their collective knowledge on dissemination and implementation designs with searches of published evaluations strategies. This article emphasizes randomized and nonrandomized designs for the traditional translational research continuum or pipeline, which builds on existing efficacy and effectiveness trials to examine how one or more evidence-based clinical/prevention interventions are adopted, scaled up, and sustained in community or service delivery systems. We also mention other designs, including hybrid designs that combine effectiveness and implementation research, quality improvement designs for local knowledge, and designs that use simulation modeling.

[1]  David L Buckeridge,et al.  Public Health Surveillance Systems: Recent Advances in Their Use and Evaluation. , 2017, Annual review of public health.

[2]  Molly A. Hall,et al.  Informatics and Data Analytics to Support Exposome-Based Discovery for Public Health. , 2017, Annual review of public health.

[3]  S. Katikireddi,et al.  Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research , 2017, Annual review of public health.

[4]  Ross C. Brownson,et al.  A review of policy dissemination and implementation research funded by the National Institutes of Health, 2007–2014 , 2015, Implementation Science.

[5]  David Henry,et al.  Designs for Testing Group-Based Interventions with Limited Numbers of Social Units: The Dynamic Wait-Listed and Regression Point Displacement Designs , 2015, Prevention Science.

[6]  William T Riley,et al.  Trials of Intervention Principles: Evaluation Methods for Evolving Behavioral Intervention Technologies , 2015, Journal of medical Internet research.

[7]  Lawrence A. Palinkas,et al.  Social Network Analysis for Program Implementation , 2015, PloS one.

[8]  Bibhas Chakraborty,et al.  Sequential multiple assignment randomized trial (SMART) with adaptive randomization for quality improvement in depression treatment program , 2015, Biometrics.

[9]  P. Nilsen Making sense of implementation theories, models and frameworks , 2015, Implementation Science.

[10]  F. Taxman,et al.  A new behavioral health services cascade framework for measuring unmet addiction health services needs and adolescent offenders: conceptual and measurement challenges , 2015, Addiction Science & Clinical Practice.

[11]  Julia Poncela-Casasnovas,et al.  Adoption of a High-Impact Innovation in a Homogeneous Population. , 2014, Physical review. X.

[12]  Wei Wang,et al.  Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial , 2014, Implementation Science.

[13]  Valerie B. Shapiro,et al.  Prevention System Mediation of Communities That Care Effects on Youth Outcomes , 2014, Prevention Science.

[14]  Daniel Almirall,et al.  Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program , 2014, Implementation Science.

[15]  B. Spring,et al.  Optimization of remotely delivered intensive lifestyle treatment for obesity using the Multiphase Optimization Strategy: Opt-IN study protocol. , 2014, Contemporary clinical trials.

[16]  J. Hawkins,et al.  The Relation Between Risk and Protective Factors for Problem Behaviors and Depressive Symptoms, Antisocial Behavior, and Alcohol Use in Adolescence. , 2014, Journal of community psychology.

[17]  Inbal Nahum-Shani,et al.  Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART) , 2014, Clinical trials.

[18]  J. Clarkson,et al.  Evaluating an audit and feedback intervention for reducing antibiotic prescribing behaviour in general dental practice (the RAPiD trial): a partial factorial cluster randomised trial protocol , 2014, Implementation Science.

[19]  Lisa Saldana,et al.  The stages of implementation completion for evidence-based practice: protocol for a mixed methods study , 2014, Implementation Science.

[20]  R. Abbott,et al.  Variation in the Sustained Effects of the Communities That Care Prevention System on Adolescent Smoking, Delinquency, and Violence , 2014, Prevention Science.

[21]  A. D. Diez Roux,et al.  Commentary: Generating rigorous evidence for public health: the need for new thinking to improve research and practice. , 2014, Annual review of public health.

[22]  R. Abbott,et al.  Youth problem behaviors 8 years after implementing the communities that care prevention system: a community-randomized trial. , 2014, JAMA pediatrics.

[23]  Linda M Collins,et al.  Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST) , 2014, Translational behavioral medicine.

[24]  Enola K Proctor,et al.  Implementation strategies: recommendations for specifying and reporting , 2013, Implementation Science.

[25]  D. Karalis,et al.  Effect of health information technology interventions on lipid management in clinical practice: a systematic review of randomized controlled trials. , 2013, Journal of clinical lipidology.

[26]  Geoff Sutcliffe,et al.  A Computational Future for Preventing HIV in Minority Communities: How Advanced Technology Can Improve Implementation of Effective Programs , 2013, Journal of acquired immune deficiency syndromes.

[27]  J. Grimshaw,et al.  Best strategies to implement clinical pathways in an emergency department setting: study protocol for a cluster randomized controlled trial , 2013, Implementation Science.

[28]  Andre Charlett,et al.  The Feedback Intervention Trial (FIT) — Improving Hand-Hygiene Compliance in UK Healthcare Workers: A Stepped Wedge Cluster Randomised Controlled Trial , 2012, PloS one.

[29]  S. Murphy,et al.  Experimental design and primary data analysis methods for comparing adaptive interventions. , 2012, Psychological methods.

[30]  M. Dennis,et al.  Using pay for performance to improve treatment implementation for adolescent substance use disorders: results from a cluster randomized trial. , 2012, Archives of pediatrics & adolescent medicine.

[31]  S. Shortreed,et al.  Questions about quitting (Q2): design and methods of a Multiphase Optimization Strategy (MOST) randomized screening experiment for an online, motivational smoking cessation intervention. , 2012, Contemporary clinical trials.

[32]  David A. Chambers,et al.  National Institutes of Health approaches to dissemination and implementation science: current and future directions. , 2012, American journal of public health.

[33]  Sara J. Czaja,et al.  Design and analysis in dissemination and implementation research , 2012 .

[34]  Enola K. Proctor,et al.  A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health , 2012, Medical care research and review : MCRR.

[35]  S. Murphy,et al.  A "SMART" design for building individualized treatment sequences. , 2012, Annual review of clinical psychology.

[36]  Ross C. Brownson,et al.  Developing the Terminology for Dissemination and Implementation Research , 2012 .

[37]  Ross C. Brownson,et al.  Dissemination and implementation research in health : translating science to practice , 2012 .

[38]  Geoffrey M. Curran,et al.  Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact , 2012, Medical care.

[39]  John J. Dziak,et al.  Multilevel factorial experiments for developing behavioral interventions: power, sample size, and resource considerations. , 2012, Psychological methods.

[40]  Edward A. Smith,et al.  Translational Research in South Africa: Evaluating Implementation Quality Using a Factorial Design , 2012, Child & youth care forum.

[41]  J. Graham,et al.  Translational Research in South Africa: Evaluating Implementation Quality Using a Factorial Design , 2012, Child & Youth Care Forum.

[42]  Guosheng Yin,et al.  Fundamentals of Clinical Trials , 2012 .

[43]  Lisa Saldana,et al.  Observational measure of implementation progress in community based settings: The Stages of implementation completion (SIC) , 2011, Implementation science : IS.

[44]  W. LeBlanc,et al.  Reconsidering low-dose aspirin therapy for cardiovascular disease: a study protocol for physician and patient behavioral change , 2011, Implementation science : IS.

[45]  Russell E Glasgow,et al.  A proposal to speed translation of healthcare research into practice: dramatic change is needed. , 2011, American journal of preventive medicine.

[46]  H. Perl Addicted to discovery: Does the quest for new knowledge hinder practice improvement? , 2011, Addictive behaviors.

[47]  Megan E. Piper,et al.  The Multiphase Optimization Strategy for Engineering Effective Tobacco Use Interventions , 2011, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[48]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[49]  M. Greenberg,et al.  Six-Year Sustainability of Evidence-Based Intervention Implementation Quality by Community-University Partnerships: The PROSPER Study , 2011, American journal of community psychology.

[50]  David V Glidden,et al.  Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. , 2010, The New England journal of medicine.

[51]  G. Aarons,et al.  Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[52]  J. King,et al.  Evaluation of a rapid HIV testing initiative in an urban, hospital-based dental clinic. , 2010, AIDS patient care and STDs.

[53]  Lawrence A. Palinkas,et al.  Mixed Method Designs in Implementation Research , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[54]  Megan E. Patrick,et al.  Influencing Adolescent Leisure Motivation: Intervention Effects of Health Wise South Africa , 2010, Journal of leisure research.

[55]  G. Aarons,et al.  Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice , 2009, Implementation science : IS.

[56]  D. Katz,et al.  A before-after implementation trial of smoking cessation guidelines in hospitalized veterans , 2009, Implementation science : IS.

[57]  Wei Wang,et al.  Implementation Science Study Protocol for a Group Randomized Controlled Trial of a Classroom-based Intervention Aimed at Preventing Early Risk Factors for Drug Abuse: Integrating Effectiveness and Implementation Research , 2022 .

[58]  Runze Li,et al.  Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs. , 2009, Psychological methods.

[59]  T. Boat,et al.  Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities , 2009 .

[60]  Simon Coulton,et al.  Screening and brief interventions for hazardous alcohol use in accident and emergency departments : a randomised controlled trial protocol , 2015 .

[61]  B. Muthén,et al.  Adaptive designs for randomized trials in public health. , 2009, Annual review of public health.

[62]  Lisette Schoonhoven,et al.  Nursing implementation science: how evidence-based nursing requires evidence-based implementation. , 2008, Journal of nursing scholarship : an official publication of Sigma Theta Tau International Honor Society of Nursing.

[63]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[64]  E. Hahn,et al.  Smoke-free laws and adult smoking prevalence. , 2008, Preventive medicine.

[65]  Wei Wang,et al.  Methods for testing theory and evaluating impact in randomized field trials: intent-to-treat analyses for integrating the perspectives of person, place, and time. , 2008, Drug and alcohol dependence.

[66]  Jeanne M. Poduska,et al.  Impact of the Good Behavior Game, a universal classroom-based behavior intervention, on young adult service use for problems with emotions, behavior, or drugs or alcohol. , 2008, Drug and alcohol dependence.

[67]  K. Kapur,et al.  Mixed‐effects Poisson regression analysis of adverse event reports: The relationship between antidepressants and suicide , 2008, Statistics in medicine.

[68]  Tihomir Asparouhov,et al.  Cluster randomized trials with treatment noncompliance. , 2008, Psychological methods.

[69]  Lisa Saldana,et al.  Engaging and Recruiting Counties in an Experiment on Implementing Evidence-based Practice in California , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[70]  Heidemarie K. Laurent,et al.  Prevention of Behavior Problems for Children in Foster Care: Outcomes and Mediation Effects , 2008, Prevention Science.

[71]  A. Kilbourne,et al.  Implementing evidence-based interventions in health care: application of the replicating effective programs framework , 2007, Implementation science : IS.

[72]  J. Mann,et al.  Early evidence on the effects of regulators' suicidality warnings on SSRI prescriptions and suicide in children and adolescents. , 2007, The American journal of psychiatry.

[73]  Shawna L. Mercer,et al.  Study designs for effectiveness and translation research :identifying trade-offs. , 2007, American journal of preventive medicine.

[74]  R. Abbott,et al.  Effects of Communities That Care on Prevention Services Systems: Findings From the Community Youth Development Study at 1.5 Years , 2007, Prevention Science.

[75]  S. Murphy,et al.  Developing adaptive treatment strategies in substance abuse research. , 2007, Drug and alcohol dependence.

[76]  H. Brenner,et al.  Effects of practitioner education, practitioner payment and reimbursement of patients’ drug costs on smoking cessation in primary care: a cluster randomised trial , 2007, Tobacco Control.

[77]  Robert D Gibbons,et al.  The role of randomized trials in testing interventions for the prevention of youth suicide , 2007, International review of psychiatry.

[78]  R. Lilford,et al.  Bmc Medical Research Methodology Open Access the Stepped Wedge Trial Design: a Systematic Review , 2022 .

[79]  Jing Guo,et al.  Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide , 2006, Clinical trials.

[80]  David M Murray,et al.  Analysis strategies for a community trial to reduce adolescent ATOD use: a comparison of random coefficient and ANOVA/ANCOVA models. , 2006, Contemporary clinical trials.

[81]  Russell E. Glasgow,et al.  Evaluating the Relevance, Generalization, and Applicability of Research , 2006, Evaluation & the health professions.

[82]  Paul A Estabrooks,et al.  Practical Clinical Trials for Translating Research to Practice: Design and Measurement Recommendations , 2005, Medical care.

[83]  G. Robert,et al.  Diffusion of innovations in service organizations: systematic review and recommendations. , 2004, The Milbank quarterly.

[84]  Susan A. Murphy,et al.  A Conceptual Framework for Adaptive Preventive Interventions , 2004, Prevention Science.

[85]  Delbert S. Elliott,et al.  Issues in Disseminating and Replicating Effective Prevention Programs , 2004, Prevention Science.

[86]  Jonathan L. Blitstein,et al.  Design and analysis of group-randomized trials: a review of recent methodological developments. , 2004, American journal of public health.

[87]  P. Chamberlain Treating Chronic Juvenile Offenders: Advances Made Through the Oregon Multidimensional Treatment Foster Care Model , 2003 .

[88]  E. Rogers,et al.  Diffusion of innovations , 1964, Encyclopedia of Sport Management.

[89]  N. Steen,et al.  Is untargeted outreach visiting in primary care effective? A pragmatic randomized controlled trial. , 2001, Journal of public health medicine.

[90]  Alexander C. Wagenaar,et al.  The Value of Interrupted Time-Series Experiments for Community Intervention Research , 2000, Prevention Science.

[91]  C. Brown,et al.  Principles for Designing Randomized Preventive Trials in Mental Health: An Emerging Developmental Epidemiology Paradigm , 1999, American journal of community psychology.

[92]  T. Coates,et al.  Mobilizing young gay and bisexual men for HIV prevention: a two-community study. , 1999, AIDS.

[93]  R. Glasgow,et al.  Evaluating the public health impact of health promotion interventions: the RE-AIM framework. , 1999, American journal of public health.

[94]  David M. Murray,et al.  Design and Analysis of Group- Randomized Trials , 1998 .

[95]  S. Raudenbush Statistical analysis and optimal design for cluster randomized trials , 1997 .

[96]  D. Sackett,et al.  Evidence based medicine: what it is and what it isn't , 1996, BMJ.

[97]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[98]  B. Flay Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. , 1986, Preventive medicine.

[99]  Howard S. Bloom,et al.  Accounting for No-Shows in Experimental Evaluation Designs , 1984 .

[100]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[101]  R. Levy The National Heart, Lung, and Blood Institute. Overview 1980. The Director's Report to the NHLBI Advisory Council. , 1982, Circulation.

[102]  D. Rubin Estimating causal effects of treatments in randomized and nonrandomized studies. , 1974 .

[103]  G. Patterson Interventions for boys with conduct problems: multiple settings, treatments, and criteria. , 1974, Journal of consulting and clinical psychology.

[104]  D M Baer,et al.  Some current dimensions of applied behavior analysis. , 1968, Journal of applied behavior analysis.

[105]  D. Campbell,et al.  Regression-Discontinuity Analysis: An Alternative to the Ex-Post Facto Experiment , 1960 .

[106]  A. B. Hill,et al.  Principles of Medical Statistics , 1950, The Indian Medical Gazette.

[107]  J. I The Design of Experiments , 1936, Nature.

[108]  C. Brown,et al.  Brief Strategic Family Therapy Treatment for Behavior Problem Youth : Theory, Intervention, Research, and Implementation 1 , 2015 .

[109]  Naihua Duan,et al.  Design of implementation studies for quality improvement programs: an effectiveness-cost-effectiveness framework. , 2014, American journal of public health.

[110]  Yinglin Xia,et al.  Observational Measures of Implementer Fidelity for a School-Based Preventive Intervention: Development, Reliability, and Validity , 2014, Prevention Science.

[111]  Ross C. Brownson,et al.  Dissemination and implementation research in health : translating science to practice , 2012 .

[112]  K. Shojania,et al.  Evidence-based quality improvement: the state of the science. , 2005, Health affairs.

[113]  S. Piantadosi Clinical Trials : A Methodologic Perspective , 2005 .

[114]  E. Sogolow,et al.  Replicating effective programs: HIV/AIDS prevention technology transfer. , 2000, AIDS education and prevention : official publication of the International Society for AIDS Education.

[115]  A FRAMEWORK FOR DEVELOPMENT AND EVALUATION OF RCTs FOR COMPLEX INTERVENTIONS TO IMPROVE HEALTH , 2000 .

[116]  R. Stuart Violent behavior : social learning approaches to prediction, management, and treatment , 1981 .