Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation

A decade ago, the Society of Prevention Research (SPR) endorsed a set of standards for evidence related to research on prevention interventions. These standards (Flay et al., Prevention Science 6:151–175, 2005) were intended in part to increase consistency in reviews of prevention research that often generated disparate lists of effective interventions due to the application of different standards for what was considered to be necessary to demonstrate effectiveness. In 2013, SPR’s Board of Directors decided that the field has progressed sufficiently to warrant a review and, if necessary, publication of “the next generation” of standards of evidence. The Board convened a committee to review and update the standards. This article reports on the results of this committee’s deliberations, summarizing changes made to the earlier standards and explaining the rationale for each change. The SPR Board of Directors endorses “The Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation.”

[1]  B. Muthén,et al.  Growth modeling with nonignorable dropout: alternative analyses of the STAR*D antidepressant trial. , 2011, Psychological methods.

[2]  Kosuke Imai,et al.  Statistical analysis of randomized experiments with non‐ignorable missing binary outcomes: an application to a voting experiment , 2009 .

[3]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[4]  Linda M. Collins,et al.  Replication in Prevention Science , 2011, Prevention Science.

[5]  R. Bosker Boekbespreking van "A.S. Bryk & S.W. Raudenbusch - Hierarchical linear models: Applications and data analysis methods" : Sage Publications, Newbury Parki, London/New Delhi 1992 , 1995 .

[6]  D. DeGarmo,et al.  Sustaining Fidelity Following the Nationwide PMTO™ Implementation in Norway , 2011, Prevention Science.

[7]  Karen A. Blase,et al.  Implementation Research: A Synthesis of the , 2005 .

[8]  G. Aarons,et al.  The role of organizational processes in dissemination and implementation research , 2017 .

[9]  Karen M. Emmons,et al.  Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination , 2012 .

[10]  Kimberly Hoagwood,et al.  Research ethics for mental health science involving ethnic minority children and youths , 2002 .

[11]  M. Greenberg,et al.  Impact Challenges in Community Science-with-Practice: Lessons from PROSPER on Transformative Practitioner-Scientist Partnerships and Prevention Infrastructure Development , 2011, American journal of community psychology.

[12]  J. Ioannidis Why Science Is Not Necessarily Self-Correcting , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[13]  Alan Ruby,et al.  Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children. Report from the Social and Character Development Research Program. NCER 2011-2001. , 2010 .

[14]  Diana L. Schroeder,et al.  The Implementation of a Statewide Bullying Prevention Program , 2012, Health promotion practice.

[15]  B. Flay Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. , 1986, Preventive medicine.

[16]  M. Eisner,et al.  The Effectiveness of Two Universal Preventive Interventions in Reducing Children's Externalizing Behavior: A Cluster Randomized Controlled Trial , 2011, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[17]  Karen A. Blase,et al.  Implementation research: A synthesis of the literature (No. 231). University of South Florida, Louis de la Parte Florida Mental Health Implementation Research Network. , 2012 .

[18]  Susan T. Ennett,et al.  A Comparison of Current Practice in School-Based Substance Use Prevention Programs with Meta-Analysis Findings , 2003, Prevention Science.

[19]  Peter M. Steiner,et al.  Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments , 2008 .

[20]  Ross C. Brownson,et al.  Developing the Terminology for Dissemination and Implementation Research , 2012 .

[21]  William M. K. Trochim,et al.  Research Design for Program Evaluation: The Regression-Discontinuity Approach , 1984 .

[22]  D. Mackinnon Introduction to Statistical Mediation Analysis , 2008 .

[23]  J. Durlak,et al.  Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation , 2008, American journal of community psychology.

[24]  Camilla A. Heid,et al.  Third Grade Follow-Up to the Head Start Impact Study: Final Report. OPRE Report 2012-45b. Executive Summary. , 2012 .

[25]  Don C Des Jarlais,et al.  Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. , 2004, American journal of public health.

[26]  Thomas D. Cook,et al.  How Bias Reduction Is Affected by Covariate Choice, Unreliability, and Mode of Data Analysis: Results From Two Types of Within-Study Comparisons , 2009, Multivariate behavioral research.

[27]  E. Foster,et al.  Estimating the Costs of Preventive Interventions , 2007, Evaluation review.

[28]  Estrogen Plus Progestin and the Incidence of Dementia and Mild Cognitive Impairment in Postmenopausal Women. The Women???s Health Initiative Memory Study: A Randomized, Controlled Trial , 2003 .

[29]  E. Mayo-Wilson,et al.  Protocol for CONSORT-SPI: an extension for social and psychological interventions , 2013, Implementation Science.

[30]  Matt Anderson,et al.  European Monitoring Centre for Drugs and Drug Addiction , 2014 .

[31]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, BMC medicine.

[32]  Craig K Enders,et al.  Missing not at random models for latent growth curve analyses. , 2011, Psychological methods.

[33]  Wei Wang,et al.  Methods for testing theory and evaluating impact in randomized field trials: intent-to-treat analyses for integrating the perspectives of person, place, and time. , 2008, Drug and alcohol dependence.

[34]  E. Mayo-Wilson,et al.  Reporting Quality of Social and Psychological Intervention Trials: A Systematic Review of Reporting Guidelines and Trial Publications , 2013, PloS one.

[35]  Brian A. Nosek,et al.  Scientific Utopia , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[36]  B. Orobio de Castro,et al.  Effectiveness of the Incredible Years parent training to modify disruptive and prosocial child behavior: a meta-analytic review. , 2013, Clinical psychology review.

[37]  Robert F. Boruch,et al.  Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination , 2005, Prevention Science.

[38]  T. Valente,et al.  Type II Translation , 2006, Evaluation & the health professions.

[39]  Evaluating Public Health Law Using Randomized Experiments , 2012 .

[40]  Michael W. Arthur,et al.  Translational Research in Action: Implementation of the Communities That Care Prevention System in 12 Communities. , 2009, Journal of community psychology.

[41]  T. Bewley,et al.  Advances in Econometrics , 1995 .

[42]  John E. Hunter,et al.  The Desperate Need for Replications , 2001 .

[43]  Stephen M. Scariano,et al.  The effects of violations of independence assumptions in the one-way ANOVA , 1987 .

[44]  G. Piaggio,et al.  Consort 2010 statement: extension to cluster randomised trials , 2012, BMJ : British Medical Journal.

[45]  S. Wolchik,et al.  Preventive Interventions for Children of Divorce , 2004 .

[46]  F. Mosteller,et al.  Evidence matters : randomized trials in education research , 2002 .

[47]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[48]  Kenneth W Griffin,et al.  Long-Term Follow-Up Effects of a School-Based Drug Abuse Prevention Program on Adolescent Risky Driving , 2004, Prevention Science.

[49]  Matthew C. Makel,et al.  Replications in Psychology Research , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[50]  Frank Fischer,et al.  The Experimenting Society: Essays in Honor of Donald T. Campbell , 1999 .

[51]  D. Hallfors,et al.  Will the 'principles of effectiveness' improve prevention practice? Early findings from a diffusion study. , 2002, Health education research.

[52]  Juan-José Díaz,et al.  An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator , 2005, The Journal of Human Resources.

[53]  R. Boruch Preface: Better Evaluation for Evidence-Based Policy: Place Randomized Trials in Education, Criminology, Welfare, and Health , 2005 .

[54]  D. Hedeker,et al.  Random-effects regression models for clustered data with an example from smoking prevention research. , 1994, Journal of consulting and clinical psychology.

[55]  Patricia J. Mrazek,et al.  Reducing risks for mental disorders: Frontiers for preventive intervention research. , 1994 .

[56]  Geoffrey M. Curran,et al.  Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact , 2012, Medical care.

[57]  R. Berk,et al.  Place Randomized Trials , 2010 .

[58]  A. Wagenaar,et al.  Public Health Law Research: Theory and Methods , 2014 .

[59]  Thomas D. Cook,et al.  Examining the Internal Validity and Statistical Precision of the Comparative Interrupted Time Series Design by Comparison With a Randomized Experiment , 2014 .

[60]  H. Pashler,et al.  Editors’ Introduction to the Special Section on Replicability in Psychological Science , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[61]  Anthony Petrosino,et al.  The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research , 2005 .

[62]  Lisa Saldana,et al.  Engaging and Recruiting Counties in an Experiment on Implementing Evidence-based Practice in California , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[63]  S. Sussman,et al.  One-year follow-up evaluation of the Project Towards No Drug Abuse (TND) dissemination trial. , 2010, Preventive medicine.

[64]  H. Bloom,et al.  Using Experiments to Assess Nonexperimental Comparison-Group Methods for Measuring Program Effects. , 2005 .

[65]  Howard S. Bloom,et al.  Learning More from Social Experiments , 2006 .

[66]  F. Gardner,et al.  Randomised controlled trial of a parenting intervention in the voluntary sector for reducing child conduct problems: outcomes and mechanisms of change. , 2006, Journal of child psychology and psychiatry, and allied disciplines.

[67]  Francis X. Diebold,et al.  Unit roots in economic time series: a selective survey , 1988 .

[68]  J. Schafer,et al.  Missing data: our view of the state of the art. , 2002, Psychological methods.

[69]  E. Stuart,et al.  The Quantum Opportunity Program Demonstration: Final Impacts. , 2006 .

[70]  D. Gottfredson,et al.  Quality of School-Based Prevention Programs: Results from a National Survey , 2002 .

[71]  Steven Glazerman,et al.  Nonexperimental Versus Experimental Estimates of Earnings Impacts , 2003 .

[72]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[73]  B. Plummer,et al.  Six-year follow-up of preventive interventions for children of divorce: a randomized controlled trial. , 2002, JAMA.

[74]  R. Edwards,et al.  Parenting intervention in Sure Start services for children at risk of developing conduct disorder: pragmatic randomised controlled trial , 2007, BMJ : British Medical Journal.

[75]  Paul Montgomery,et al.  The Oxford Implementation Index: a new tool for incorporating implementation data into systematic reviews and meta-analyses. , 2013, Journal of clinical epidemiology.

[76]  D. Altman,et al.  CONSORT statement: extension to cluster randomised trials , 2004, BMJ : British Medical Journal.

[77]  D. A. Kenny,et al.  Consequences of violating the independence assumption in analysis of variance. , 1986 .

[78]  E. Mayo-Wilson,et al.  The need for new guidelines to improve the reporting of trials in child and adolescent mental health. , 2013, Journal of child psychology and psychiatry, and allied disciplines.

[79]  David A. Chambers,et al.  The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change , 2013, Implementation Science.

[80]  R. Flewelling,et al.  The Effects of Project ALERT One Year Past Curriculum Completion , 2010, Prevention Science.

[81]  Vandana Sundaram,et al.  The American Journal of Public Health. , 1945, American journal of public health and the nation's health.

[82]  E. Mayo-Wilson,et al.  Developing a reporting guideline for social and psychological intervention trials , 2013, The British journal of psychiatry : the journal of mental science.

[83]  Huey-tsyh Chen Theory-driven evaluations , 1990 .

[84]  Jens Ludwig,et al.  Long‐Term Effects of Head Start on Low‐Income Children , 2008, Annals of the New York Academy of Sciences.

[85]  C. H. Brown,et al.  Statistical methods for preventive trials in mental health. , 1993, Statistics in medicine.

[86]  J. Ioannidis Contradicted and Initially Stronger Effects in Highly Cited Clinical Research , 2005 .

[87]  Joann L. Robinson,et al.  Effects of Home Visits by Paraprofessionals and by Nurses: Age 4 Follow-Up Results of a Randomized Trial , 2004, Pediatrics.

[88]  Coady Wing,et al.  STRENGTHENING THE REGRESSION DISCONTINUITY DESIGN USING ADDITIONAL DESIGN ELEMENTS: A WITHIN‐STUDY COMPARISON , 2013 .

[89]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[90]  A C Wagenaar,et al.  Preventing injuries to children through compulsory automobile safety seat use. , 1986, Pediatrics.

[91]  Hyung Min Kim,et al.  Efficacy vs effectiveness trial results of an indicated "model" substance abuse program: implications for public health. , 2006, American journal of public health.

[92]  John F. Steiner,et al.  Comparative Effectiveness Research to Accelerate Translation: Recommendations for an Emerging Field of Science , 2012 .

[93]  Michael L Bloomquist,et al.  Going-to-scale with the Early Risers conduct problems prevention program: use of a comprehensive implementation support (CIS) system to optimize fidelity, participation and child outcomes. , 2013, Evaluation and program planning.

[94]  Louise A. Rohrbach,et al.  Addressing Core Challenges for the Next Generation of Type 2 Translation Research and Systems: The Translation Science to Population Impact (TSci Impact) Framework , 2013, Prevention Science.

[95]  Delbert S. Elliott,et al.  Issues in Disseminating and Replicating Effective Prevention Programs , 2004, Prevention Science.

[96]  H. Pashler,et al.  Is the Replicability Crisis Overblown? Three Arguments Examined , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[97]  David M. Murray,et al.  Design and Analysis of Group- Randomized Trials , 1998 .

[98]  P. Albert,et al.  Models for longitudinal data: a generalized estimating equation approach. , 1988, Biometrics.

[99]  Lauren H. Supplee,et al.  Introduction to the Special Issue: Subgroup Analysis in Prevention and Intervention Research , 2013, Prevention Science.

[100]  Ross C. Brownson,et al.  Dissemination and implementation research in health : translating science to practice , 2012 .

[101]  William M. K. Trochim,et al.  Research methods knowledge base , 2001 .

[102]  Gracelyn Cruden,et al.  Advancing Science Through Collaborative Data Sharing and Synthesis , 2013, Perspectives on psychological science : a journal of the Association for Psychological Science.

[103]  Anthony L. Hemmelgarn,et al.  Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. , 2010, Journal of consulting and clinical psychology.

[104]  M. Greenberg,et al.  Six-Year Sustainability of Evidence-Based Intervention Implementation Quality by Community-University Partnerships: The PROSPER Study , 2011, American journal of community psychology.

[105]  Robert B Wallace,et al.  Estrogen plus progestin and the incidence of dementia and mild cognitive impairment in postmenopausal women: the Women's Health Initiative Memory Study: a randomized controlled trial. , 2003, JAMA.

[106]  David B. Wilson,et al.  The Strengthening Washington D.C. Families Project: A Randomized Effectiveness Trial of Family-Based Prevention , 2006, Prevention Science.

[107]  Camilla A. Heid,et al.  Head Start Impact Study. Final Report. , 2010 .

[108]  W. Greene,et al.  计量经济分析 = Econometric analysis , 2009 .