Challenges Facing Evidence-Based Prevention: Incorporating an Abductive Theory of Method

Current systems used to determine whether prevention programs are “evidence-based” rely on the logic of deductive reasoning. This reliance has fostered implementation of strategies with explicitly stated evaluation criteria used to gauge program validity and suitability for dissemination. Frequently, investigators resort to the randomized controlled trial (RCT) combined with null hypothesis significance testing (NHST) as a means to rule out competing hypotheses and determine whether an intervention works. The RCT design has achieved success across numerous disciplines but is not without limitations. We outline several issues that question allegiance to the RCT, NHST, and the hypothetico-deductive method of scientific inquiry. We also discuss three challenges to the status of program evaluation including reproducibility, generalizability, and credibility of findings. As an alternative, we posit that extending current program evaluation criteria with principles drawn from an abductive theory of method (ATOM) can strengthen our ability to address these challenges and advance studies of drug prevention. Abductive reasoning involves working from observed phenomena to the generation of alternative explanations for the phenomena and comparing the alternatives to select the best possible explanation. We conclude that an ATOM can help increase the influence and impact of evidence-based prevention for population benefit.

[1]  Dennis M. Gorman,et al.  The decline effect in evaluations of the impact of the Strengthening Families Program for Youth 10-14 (SFP 10-14) on adolescent substance use , 2017 .

[2]  Peter Lipton,et al.  Inference to the best explanation , 1993 .

[3]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[4]  T. Levine,et al.  A Critical Assessment of Null Hypothesis Significance Testing in Quantitative Communication Research , 2008 .

[5]  B. Thompson,et al.  How to Estimate and Interpret Various Effect Sizes , 2004 .

[6]  Igor Douven.  Stanford Encyclopedia of Philosophy , 2011 .

[7]  S. Ennett,et al.  Factors associated with adoption of evidence-based substance use prevention curricula in US school districts. , 2005, Health education research.

[8]  J. Durlak,et al.  Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation , 2008, American journal of community psychology.

[9]  Jacob Stegenga,et al.  Is meta-analysis the platinum standard of evidence? , 2011, Studies in history and philosophy of biological and biomedical sciences.

[10]  Thomas D. Cook,et al.  Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research in Prevention Science: Next Generation , 2015, Prevention Science.

[11]  D. Sharpe.  Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research. , 2004 .

[12]  David Ogilvie,et al.  Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go? , 2005, Journal of Epidemiology and Community Health.

[13]  Mary Jane Rotheram-Borus,et al.  Next generation of preventive interventions. , 2003, Journal of the American Academy of Child and Adolescent Psychiatry.

[14]  Jerzy Neyman,et al.  "Inductive Behavior" as a Basic Concept of Philosophy of Science , 1957 .

[15]  John P. A. Ioannidis,et al.  The Power of Bias in Economics Research , 2017 .

[16]  D. Mackinnon,et al.  The effect of marriage on young adult heavy drinking and its mediators: results from two methods of adjusting for selection into marriage. , 2010, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[17]  E. J. Capaldi,et al.  Are theories to be evaluated in isolation or relative to alternatives? An abductive view. , 2008, The American journal of psychology.

[18]  D. Gorman.  Drug and violence prevention: Rediscovering the critical rational dimension of evaluation research , 2005 .

[19]  S. Ennett,et al.  The Prevalence of Evidence-Based Drug Use Prevention Curricula in U.S. Middle Schools in 2005 , 2009, Prevention Science.

[20]  David Henry,et al.  Alternatives to Randomized Control Trial Designs for Community-Based Prevention Evaluation , 2017, Prevention Science.

[21]  C. Hempel,et al.  A Definition of "Degree of Confirmation" , 1945, Philosophy of Science.

[22]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, BMC medicine.

[23]  Samantha F. Anderson,et al.  There's more than one way to conduct a replication study: Beyond statistical significance. , 2016, Psychological methods.

[24]  Nancy Cartwright,et al.  Are RCTs the Gold Standard? , 2007 .

[25]  S. Ennett,et al.  The Prevalence of Evidence-based Drug Use Prevention Curricula in U.S. Middle Schools in 2008 , 2011, Prevention Science.

[26]  Robert F. Boruch,et al.  Standards of Evidence: Criteria for Efficacy, Effectiveness and Dissemination , 2005, Prevention Science.

[27]  M. Stemmler,et al.  Experiences in Disseminating Evidence-Based Prevention Programs in a Real-World Setting , 2015, Prevention Science.

[28]  Jing Guo,et al.  Dynamic wait-listed designs for randomized trials: new designs for prevention of youth suicide , 2006, Clinical trials.

[29]  Sandee J. Kyler,et al.  Sustaining Evidence-based Interventions Under Real-world Conditions: Results from a Large-scale Diffusion Project , 2010, Prevention Science.

[30]  K. Emmons,et al.  How can we increase translation of research into practice? Types of evidence needed. , 2007, Annual review of public health.

[31]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, Journal of clinical epidemiology.

[32]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[33]  William Whewell,et al.  The philosophy of the inductive sciences , 1847 .

[34]  S. Kellam,et al.  A Framework for Understanding “Evidence” in Prevention Research and Programs , 2003, Prevention Science.

[35]  B. Haig,et al.  An abductive perspective on clinical reasoning and case formulation. , 2008, Journal of clinical psychology.

[36]  N. Jacobson,et al.  Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. , 1991, Journal of consulting and clinical psychology.

[37]  Gerry Richardson,et al.  Effectiveness of a nurse-led intensive home-visitation programme for first-time teenage mothers (Building Blocks): a pragmatic randomised controlled trial , 2016, The Lancet.

[38]  B. Haig.  An abductive theory of scientific method. , 2005, Psychological methods.

[39]  Julian Reiss,et al.  A Pragmatist Theory of Evidence , 2015, Philosophy of Science.

[40]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[42]  Scott D. Gest,et al.  Effects of PROSPER on the influence potential of prosocial versus antisocial youth in adolescent friendship networks. , 2013, The Journal of adolescent health : official publication of the Society for Adolescent Medicine.

[43]  Paul Thagard,et al.  The Best Explanation: Criteria for Theory Choice , 1978 .

[44]  D. Gorman.  The best of practices, the worst of practices: The making of science-based primary prevention programs. , 2003, Psychiatric services.

[45]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[46]  David J. Kupfer,et al.  Size of Treatment Effects and Their Importance to Clinical Research and Practice , 2006, Biological Psychiatry.

[47]  K. Popper,et al.  Conjectures and refutations;: The growth of scientific knowledge , 1972 .

[48]  Jan Sprenger,et al.  A Synthesis of Hempelian and Hypothetico-Deductive Confirmation , 2013 .

[49]  S. Maxwell.  The persistence of underpowered studies in psychological research: causes, consequences, and remedies. , 2004, Psychological methods.

[50]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[51]  Ross E. G. Upshur,et al.  Randomized controlled trials in the West African Ebola virus outbreak , 2016, Clinical trials.

[52]  Kenneth W Griffin,et al.  Prevention science, drug abuse prevention, and Life Skills Training: Comments on the state of the science , 2005 .

[53]  B. Osimani.  Until RCT proven? On the asymmetry of evidence requirements for risk assessment. , 2013, Journal of evaluation in clinical practice.

[54]  Catherine P. Bradshaw,et al.  Assessing the Generalizability of Randomized Trial Results to Target Populations , 2015, Prevention Science.

[55]  Linda M. Collins,et al.  Replication in Prevention Science , 2011, Prevention Science.

[56]  Daniel G Campos,et al.  The Logic of Medical Diagnosis , 2013, Perspectives in biology and medicine.

[57]  R. Glasgow,et al.  Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. , 2003, American journal of public health.

[58]  David R. Holtgrave,et al.  Alternatives to the randomized controlled trial. , 2008, American journal of public health.

[59]  R. Fisher,et al.  The Logic of Inductive Inference , 1935 .

[60]  D. Gorman,et al.  The creation of evidence in 'evidence-based' drug prevention: a critique of the Strengthening Families Program Plus Life Skills Training evaluation. , 2007, Drug and alcohol review.

[61]  J. Carlin,et al.  Beyond Power Calculations , 2014, Perspectives on psychological science : a journal of the Association for Psychological Science.

[62]  C. Brown,et al.  Principles for Designing Randomized Preventive Trials in Mental Health: An Emerging Developmental Epidemiology Paradigm , 1999, American journal of community psychology.

[63]  Geoffrey M. Curran,et al.  Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact , 2012, Medical care.

[64]  Allison Gruner Gandhi,et al.  The Devil Is in the Details , 2007, Evaluation review.

[65]  Alexander C. Wagenaar,et al.  The Value of Interrupted Time-Series Experiments for Community Intervention Research , 2000, Prevention Science.

[66]  T. Herrenkohl,et al.  Parent and peer pathways linking childhood experiences of abuse with marijuana use in adolescence and adulthood. , 2017, Addictive behaviors.

[67]  R. Catalano,et al.  Doing prevention science: A response to Dennis M. Gorman and a brief history of the quasi-experimental study nested within the Seattle Social Development Project , 2005 .

[68]  D. Roth,et al.  Extending the evidence hierarchy to enhance evidence-based practice for substance use disorders. , 2006, Addiction.

[69]  S. Pepper.  World Hypotheses: A Study in Evidence , 1942 .

[70]  B. Haig.  On the permissiveness of the abductive theory of method. , 2008, Journal of clinical psychology.

[71]  Melanie Eisner,et al.  No effects in independent prevention trials: can we reject the cynical view? , 2009 .

[72]  S G West,et al.  The science of prevention. A conceptual framework and some directions for a national research program. , 1993, The American psychologist.

[73]  D. Gorman.  The ''science'' of drug and alcohol prevention: the case of the randomized trial of the Life Skills Training program , 2002 .

[74]  L. Harlow,et al.  Big data in psychology: Introduction to the special issue. , 2016, Psychological methods.

[75]  Charles B. Fleming,et al.  A Framework for Testing and Promoting Expanded Dissemination of Promising Preventive Interventions That Are Being Implemented in Community Settings , 2014, Prevention Science.

[76]  J. Worrall.  Evidence in Medicine and Evidence‐Based Medicine , 2007 .

[77]  Why the Resistance to Statistical Innovations? A Comment on Sharpe (2013) , 2017, Psychological methods.

[78]  Wanqing Zhang,et al.  Using Mixed Methods Effectively in Prevention Science: Designs, Procedures, and Examples , 2014, Prevention Science.

[79]  R. Lanfear,et al.  The Extent and Consequences of P-Hacking in Science , 2015, PLoS biology.

[80]  J. Brook,et al.  The psychosocial etiology of adolescent drug use: a family interactional approach. , 1990, Genetic, social, and general psychology monographs.

[81]  D. Fergusson,et al.  Tests of causal links between alcohol abuse or dependence and major depression. , 2009, Archives of general psychiatry.

[82]  D. Fanelli.  Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data , 2010, PloS one.

[83]  T. Neylan,et al.  Novel implementation research designs for scaling up global mental health care: overcoming translational challenges to address the world’s leading cause of disability , 2016, International Journal of Mental Health Systems.

[84]  D. Mayo.  Duhem's Problem, the Bayesian Way, and Error Statistics, or "What's Belief Got to Do with It?" , 1997, Philosophy of Science.

[85]  L. Collins,et al.  Seven ways to increase power without increasing N. , 1994, NIDA research monograph.

[86]  B. Haig.  Inference to the best explanation: a neglected approach to theory appraisal in psychology. , 2009, The American journal of psychology.

[87]  D. Gorman.  Is Project Towards No Drug Abuse (Project TND) an Evidence-Based Drug and Violence Prevention Program? A Review and Reappraisal of the Evaluation Studies , 2014, The Journal of Primary Prevention.

[88]  Cindy A. Crusto,et al.  What works in prevention. Principles of effective prevention programs. , 2003, The American psychologist.

[89]  Arthur W. Burks,et al.  Peirce's Theory of Abduction , 1946, Philosophy of Science.

[90]  Richard A. Heath,et al.  Nonlinear Dynamics: Techniques and Applications in Psychology , 2000 .

[91]  D. Krantz.  The Null Hypothesis Testing Controversy in Psychology , 1999 .

[92]  Ahnalee M. Brincks,et al.  Overview of meta-analyses of the prevention of mental health, substance use, and conduct problems. , 2014, Annual review of clinical psychology.

[93]  R. Rosenman,et al.  Detecting selection effects in community implementations of family-based substance abuse prevention programs. , 2010, American journal of public health.

[94]  José Szapocznik,et al.  An ecodevelopmental framework for organizing the influences on drug abuse: A developmental model of risk and protection. , 1999 .

[95]  J. Brooks.  Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[96]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[97]  Louise A. Rohrbach,et al.  Addressing Core Challenges for the Next Generation of Type 2 Translation Research and Systems: The Translation Science to Population Impact (TSci Impact) Framework , 2013, Prevention Science.

[98]  J. Shonkoff,et al.  Rethinking evidence-based practice and two-generation programs to create the future of early childhood policy , 2013, Development and Psychopathology.

[99]  Ben A. Williams,et al.  Perils of Evidence-Based Medicine , 2010, Perspectives in biology and medicine.

[100]  E. Tanner‐Smith,et al.  Meta-analysis in Prevention Science , 2014 .

[101]  P. Meehl.  Theory-Testing in Psychology and Physics: A Methodological Paradox , 1967, Philosophy of Science.

[102]  William Bechtel,et al.  Looking down, around, and up: Mechanistic explanation in psychology , 2009 .