A mixed methods multiple case study of implementation as usual in children’s social service organizations: study protocol

BackgroundImproving quality in children’s mental health and social service settings will require implementation strategies capable of moving effective treatments and other innovations (e.g., assessment tools) into routine care. It is likely that efforts to identify, develop, and refine implementation strategies will be more successful if they are informed by relevant stakeholders and are responsive to the strengths and limitations of the contexts and implementation processes identified in usual care settings. This study will describe: the types of implementation strategies used; how organizational leaders make decisions about what to implement and how to approach the implementation process; organizational stakeholders’ perceptions of different implementation strategies; and the potential influence of organizational culture and climate on implementation strategy selection, implementation decision-making, and stakeholders’ perceptions of implementation strategies.Methods/designThis study is a mixed methods multiple case study of seven children’s social service organizations in one Midwestern city in the United States that compose the control group of a larger randomized controlled trial. Qualitative data will include semi-structured interviews with organizational leaders (e.g., CEOs/directors, clinical directors, program managers) and a review of documents (e.g., implementation and quality improvement plans, program manuals, etc.) that will shed light on implementation decision-making and specific implementation strategies that are used to implement new programs and practices. Additionally, focus groups with clinicians will explore their perceptions of a range of implementation strategies. This qualitative work will inform the development of a Web-based survey that will assess the perceived effectiveness, relative importance, acceptability, feasibility, and appropriateness of implementation strategies from the perspective of both clinicians and organizational leaders. Finally, the Organizational Social Context measure will be used to assess organizational culture and climate. Qualitative, quantitative, and mixed methods data will be analyzed and interpreted at the case level as well as across cases in order to highlight meaningful similarities, differences, and site-specific experiences.DiscussionThis study is designed to inform efforts to develop more effective implementation strategies by fully describing the implementation experiences of a sample of community-based organizations that provide mental health services to youth in one Midwestern city.

[1]  L. Palinkas,et al.  Social networks and implementation of evidence-based practices in public youth-serving systems: a mixed-methods study , 2011, Implementation science : IS.

[2]  Leonard Bickman,et al.  Change What? Identifying Quality Improvement Targets by Investigating Usual Mental Health Care , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[3]  Lawrence A. Palinkas,et al.  Mixed-methods designs in mental health services research: a review. , 2011, Psychiatric services.

[4]  D. Berwick The science of improvement. , 2008, JAMA.

[5]  R. McHugh,et al.  Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. , 2009, Behaviour research and therapy.

[6]  E. Rogers,et al.  Diffusion of Innovations, 5th Edition , 2003 .

[7]  J. Weisz,et al.  More Practice, Less Preach? The Role of Supervision Processes and Therapist Characteristics in EBP Implementation , 2013, Administration and Policy in Mental Health and Mental Health Services Research.

[8]  R. Raghavan The role of economic evaluation in dissemination and implementation research , 2012 .

[9]  Abraham Wandersman,et al.  Practical Implementation Science: Developing and Piloting the Quality Implementation Tool , 2012, American journal of community psychology.

[10]  Sonja K Schoenwald,et al.  It's a Bird, It's A Plane, It's … Fidelity Measurement In the Real World. , 2011, Clinical psychology : a publication of the Division of Clinical Psychology of the American Psychological Association.

[11]  C. Dulmus,et al.  Organizational Leaders’ and Staff Members’ Appraisals of Their Work Environment Within a Children’s Social Service System , 2014, Human service organizations, management, leadership & governance.

[12]  P. Kendall,et al.  Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. , 2010, Clinical psychology : a publication of the Division of Clinical Psychology of the American Psychological Association.

[13]  Robert E. Stake,et al.  Multiple Case Study Analysis , 2005 .

[14]  Véronique Ambrosini,et al.  Using Single Respondents in Strategy Research , 1997 .

[15]  L. Solberg Guideline implementation: what the literature doesn't tell us. , 2000, The Joint Commission journal on quality improvement.

[16]  D M Berwick,et al.  A primer on leading the improvement of systems , 1996, BMJ.

[17]  Becky F. Antle,et al.  Washington State's adoption of a child welfare practice model: An illustration of the Getting To Outcomes implementation framework , 2013 .

[18]  A. Kluger,et al.  Feedback, the various tasks of the doctor, and the feedforward alternative , 2010, Medical education.

[19]  D. Chambless,et al.  Interesting practitioners in training in empirically supported treatments: research reviews versus case studies. , 2009, Journal of clinical psychology.

[20]  F. Leichsenring,et al.  Effectiveness of long-term psychodynamic psychotherapy: a meta-analysis. , 2008, Journal of the American Medical Association (JAMA).

[21]  D. Parrish,et al.  Development and Validation of the Evidence-Based Practice Process Assessment Scale: Preliminary Findings , 2010 .

[22]  K. Kelleher,et al.  Assessing the Organizational Social Context (OSC) of Mental Health Services: Implications for Research and Practice , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[23]  R. Cicerone Initial National Priorities for Comparative Effectiveness Research , 2009 .

[24]  R. Barbour Doing Focus Groups , 2008 .

[25]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[26]  Whitney Berta,et al.  Role of champions in the implementation of patient safety practice change. , 2009, Healthcare quarterly.

[27]  K. Wells,et al.  Community-based partnered research: new directions in mental health services research. , 2011, Ethnicity & disease.

[28]  G. Robert,et al.  Diffusion of innovations in service organizations: systematic review and recommendations. , 2004, The Milbank quarterly.

[29]  A. Kazdin,et al.  Comorbidity, case complexity, and effects of evidence-based treatment for children referred for disruptive behavior. , 2006, Journal of consulting and clinical psychology.

[30]  S. Shortell Increasing Value: A Research Agenda for Addressing the Managerial and Organizational Challenges Facing Health Care Delivery in the United States , 2004, Medical care research and review : MCRR.

[31]  Alicia C. Bunger,et al.  Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[32]  Jacqueline A Pugh,et al.  Clinical practice guideline implementation strategy patterns in Veterans Affairs primary care clinics. , 2007, Health services research.

[33]  M. Chin,et al.  Improving the Effectiveness of Health Care Innovation Implementation , 2013, Medical care research and review : MCRR.

[34]  A D Oxman,et al.  Changing physician performance. A systematic review of the effect of continuing medical education strategies. , 1995, JAMA.

[35]  P. Kontos,et al.  Implementation Science BioMed Central Debate , 2009 .

[36]  Robert E Drake,et al.  The role of staff turnover in the implementation of evidence-based practices in mental health care. , 2008, Psychiatric services.

[37]  Jackson A. Nickerson Leading Change in a Web 2.1 World , 2010 .

[38]  D. Kolko,et al.  The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. , 2010, Clinical psychology review.

[39]  Charles A. Rapp,et al.  Barriers to Evidence-Based Practice Implementation: Results of a Qualitative Study , 2010, Community Mental Health Journal.

[40]  David A. Chambers,et al.  Implementation Research in Mental Health Services: an Emerging Science with Conceptual, Methodological, and Training challenges , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[41]  Karen A. Blase,et al.  Implementation research: A synthesis of the literature (No. 231). University of South Florida, Louis de la Parte Florida Mental Health Implementation Research Network. , 2012 .

[42]  J Lomas,et al.  Evaluating the Message: The Relationship Between Compliance Rate and the Subject of a Practice Guideline , 1994, Medical care.

[43]  Alf Henryk Wulf,et al.  Opening plenary , 2012 .

[44]  J Sorenson,et al.  Implementation of research , 1992 .

[45]  Susan Michie,et al.  No more ‘business as usual’ with audit and feedback interventions: towards an agenda for a reinvigorated intervention , 2014, Implementation Science.

[46]  E. Balas,et al.  Managing Clinical Knowledge for Health Care Improvement , 2000, Yearbook of Medical Informatics.

[47]  Anthony L. Hemmelgarn,et al.  The Shortform Assessment for Children: An Assessment and Outcome Measure for Child Welfare and Juvenile Justice , 2002 .

[48]  StrategIc PlaN,et al.  National Institute of Mental Health Strategic Plan | 2008 , 2008 .

[49]  Thomas R Insel,et al.  Translating scientific opportunity into public health impact: a strategic plan for research on mental illness. , 2009, Archives of general psychiatry.

[50]  Patricia L. Kohl,et al.  The State of Parent Training: Program Offerings and Empirical Support , 2009, Families in society : the journal of contemporary human services.

[51]  N. Duan,et al.  Quality of publicly-funded outpatient specialty mental health care for common childhood psychiatric disorders in California. , 2005, Journal of the American Academy of Child and Adolescent Psychiatry.

[52]  G. Aarons,et al.  The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for a protective effect. , 2009, Journal of consulting and clinical psychology.

[53]  A D Oxman,et al.  Educational outreach visits: effects on professional practice and health care outcomes. , 2007, The Cochrane database of systematic reviews.

[54]  C. Glisson Assessing and Changing Organizational Culture and Climate for Effective Services , 2007 .

[55]  F. Mcmanus,et al.  Establishing evidence-based training in cognitive behavioral therapy: A review of current empirical findings and theoretical guidance. , 2010, Clinical psychology review.

[56]  David A. Chambers,et al.  Research and services partnerships: partnership: a fundamental component of dissemination and implementation research. , 2013, Psychiatric Services.

[57]  Ann F Garland,et al.  Identifying common elements of evidence-based psychosocial treatments for children's disruptive behavior problems. , 2008, Journal of the American Academy of Child and Adolescent Psychiatry.

[58]  L. Hoffman Evidence-Based Psychodynamic Psychotherapy , 2009 .

[59]  Christopher J. Miller,et al.  Community mental health provider modifications to cognitive therapy: implications for sustainability. , 2013, Psychiatric services.

[60]  Mindy E. Flanagan,et al.  Open Access Implementation Science the Effect of Provider-and Workflow-focused Strategies for Guideline Implementation on Provider Acceptance , 2022 .

[61]  D. McCarty,et al.  Employing policy and purchasing levers to increase the use of evidence-based practices in community-based substance abuse treatment settings: reports from single state authorities. , 2011, Evaluation and program planning.

[62]  D. Delprato Acceptance and Commitment Therapy: An Experiental Approach to Behavior Change , 2001 .

[63]  Jonathan Sherbino,et al.  How does an "opinion leader" influence my practice? , 2010, CJEM.

[64]  Anthony L. Hemmelgarn,et al.  Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. , 2010, Journal of consulting and clinical psychology.

[65]  John W. Creswell,et al.  Designing and Conducting Mixed Methods Research , 2006 .

[66]  M. Eccles,et al.  Planning and Studying Improvement in Patient Care: The Use of Theoretical Perspectives , 2007, The Milbank quarterly.

[67]  N Freemantle,et al.  Printed educational materials: effects on professional practice and health care outcomes. , 2000, The Cochrane database of systematic reviews.

[68]  J. Kotter,et al.  The Heart of Change: Real-Life Stories of How People Change Their Organizations , 2002 .

[69]  I. Graham,et al.  Planned action theories , 2013 .

[70]  M. Gabe Mental health: a report of the Surgeon General. , 2000, Home care provider.

[71]  Shannon Wiltsey Stirman,et al.  Development of a framework and coding system for modifications and adaptations of evidence-based interventions , 2013, Implementation Science.

[72]  Jill J. Francis,et al.  Designing theoretically-informed implementation interventions , 2006, Implementation science : IS.

[73]  Harold Alan Pincus,et al.  Can Psychiatry Cross the Quality Chasm? Improving the Quality of Health Care For Mental and Substance Use Conditions , 2011 .

[74]  B. Golden The past is the past--or is it? The use of retrospective accounts as indicators of past strategy. , 1992, Academy of Management journal. Academy of Management.

[75]  J. Pines,et al.  Risk tolerance for the exclusion of potentially life-threatening diseases in the ED. , 2007, The American journal of emergency medicine.

[76]  B MilesMatthew,et al.  Qualitative Data Analysis , 2009, Approaches and Processes of Social Science Research.

[77]  J. Weisz,et al.  Provider attitudes toward evidence-based practices: are the concerns with the evidence or with the manuals? , 2009, Psychiatric services.

[78]  Elena Parmelli,et al.  Local opinion leaders: effects on professional practice and health care outcomes. , 2011, The Cochrane database of systematic reviews.

[79]  Norbert Donner-Banzhoff,et al.  Acceptance and perceived barriers of implementing a guideline for managing low back in general practice , 2008, Implementation science : IS.

[80]  David R Flum,et al.  Risk taking and tolerance of uncertainty: implications for surgeons. , 2006, The Journal of surgical research.

[81]  Laura Johnson,et al.  How Many Interviews Are Enough? , 2006 .

[82]  S. Elo,et al.  Qualitative Content Analysis , 2014 .

[83]  K. Merikangas,et al.  Lifetime prevalence of mental disorders in U.S. adolescents: results from the National Comorbidity Survey Replication--Adolescent Supplement (NCS-A). , 2010, Journal of the American Academy of Child and Adolescent Psychiatry.

[84]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[85]  Robert D Gibbons,et al.  Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: a randomized effectiveness trial. , 2012, Archives of general psychiatry.

[86]  Kelli Scott,et al.  Using Measurement-Based Care to Enhance Any Treatment. , 2015, Cognitive and behavioral practice.

[87]  Danielle L. Fettes,et al.  Mixed Methods for Implementation Research , 2012, Child maltreatment.

[88]  M. Brommels,et al.  Implementing clinical guidelines in psychiatry: a qualitative study of perceived facilitators and barriers , 2010, BMC psychiatry.

[89]  C. Glisson,et al.  The Effects of Organizational Culture and Climate on the Access to Mental Health Care in Child Welfare and Juvenile Justice Systems , 2006, Administration and Policy in Mental Health and Mental Health Services Research.

[90]  N. Denzin The research act: A theoretical introduction to sociological methods , 1977 .

[91]  M. Marshall Sampling for qualitative research. , 1996, Family practice.

[92]  E. Tornquist,et al.  BARRIERS: the barriers to research utilization scale. , 1991, Applied nursing research : ANR.

[93]  R Brian Haynes,et al.  Strategies to enhance venous thromboprophylaxis in hospitalized medical patients (SENTRY): a pilot cluster randomized trial , 2013, Implementation Science.

[94]  L. Siminoff,et al.  Empirical methods for bioethics : a primer , 2008 .

[95]  N. Freemantle,et al.  Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? , 1999, JAMA.

[96]  Sylwia Męcfal Recenzja książki. Robert K. yin, Case Study Research. Design and Methods (fourth Edition), thousand Oaks, CA: Sage Publications, 2009 , 2012 .

[97]  J. Weisz,et al.  Empirically tested psychotherapies for youth internalizing and externalizing problems and disorders. , 2004, Child and adolescent psychiatric clinics of North America.

[98]  G. Pisano,et al.  Disrupted Routines: Team Learning and New Technology Implementation in Hospitals , 2001 .

[99]  Bruce F Chorpita,et al.  Understanding the common elements of evidence-based practice: misconceptions and clinical examples. , 2007, Journal of the American Academy of Child and Adolescent Psychiatry.

[100]  Brooks C. Holtom,et al.  Survey response rate levels and trends in organizational research , 2008 .

[101]  Eric Rice,et al.  Influence network linkages across implementation strategy conditions in a randomized controlled trial of two strategies for scaling up evidence-based practices in public youth-serving systems , 2013, Implementation Science.

[102]  G. Aarons,et al.  The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership , 2014, Implementation Science.

[103]  B. Powell,et al.  Mental Health Clinicians' Experiences of Implementing Evidence-Based Treatments , 2013, Journal of evidence-based social work.

[104]  José-Manuel Morales González,et al.  Child abuse prevention , 1997 .

[105]  Jennifer L. Magnabosco Innovations in mental health services implementation: a report on state-level data from the U.S. Evidence-Based Practices Project , 2006, Implementation science : IS.

[106]  Enola K Proctor,et al.  Advancing the application, quality and harmonization of implementation science measures , 2012, Implementation Science.

[107]  Vj. Janesick The choreography of qualitative research design , 2000 .

[108]  S. Scott,et al.  Development of a checklist to assess the quality of reporting of knowledge translation interventions using the Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations , 2013, Implementation Science.

[109]  T. Boat,et al.  Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities , 2009 .

[110]  G. Aarons,et al.  Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. , 2013, Journal of the American Academy of Child and Adolescent Psychiatry.

[111]  Jennifer I. Manuel,et al.  Preparing Social Work Practitioners to Use Evidence-Based Practice , 2009 .

[112]  D. M. Dickinson,et al.  Implementing Motivational Interviewing: Lessons from Clinical Experiences. , 2006 .

[113]  Bryan J Weiner,et al.  The meaning and measurement of implementation climate , 2011, Implementation science : IS.

[114]  Jim Collins,et al.  Good to Great - (Why Some Companies Make the Leap and others Don't) , 2009 .

[115]  David A. Chambers,et al.  The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change , 2013, Implementation Science.

[116]  Alastair Baker,et al.  Crossing the Quality Chasm: A New Health System for the 21st Century , 2001, BMJ : British Medical Journal.

[117]  L. Palinkas,et al.  Dynamic adaptation process to implement an evidence-based child maltreatment intervention , 2012, Implementation Science.

[118]  Deborah K. Padgett,et al.  Qualitative and Mixed Methods in Public Health , 2011 .

[119]  S. Straus,et al.  Lost in knowledge translation: Time for a map? , 2006, The Journal of continuing education in the health professions.

[120]  Kimberly Eaton Hoagwood,et al.  Understanding the components of quality improvement collaboratives: a systematic literature review. , 2013, The Milbank quarterly.

[121]  E. Zerhouni,et al.  Clinical and translational science awards: a framework for a national research agenda. , 2006, Translational research : the journal of laboratory and clinical medicine.

[122]  J. Grimshaw,et al.  Tailored implementation for chronic diseases (TICD): A project protocol , 2011, Implementation science : IS.

[123]  S. Marcus,et al.  Policy to implementation: evidence-based practice in community mental health – study protocol , 2013, Implementation Science.

[124]  Anthony L. Hemmelgarn,et al.  The effects of organizational climate and interorganizational coordination on the quality and outcomes of children's service systems. , 1998, Child abuse & neglect.

[125]  Cathleen Willging,et al.  Interagency Collaborative Team Model for Capacity Building to Scale-Up Evidence-Based Practice. , 2014, Children and youth services review.

[126]  Fredric Wolf,et al.  Continuing education meetings and workshops: effects on professional practice and health care outcomes. , 2009, The Cochrane database of systematic reviews.

[127]  C. Bright,et al.  Toward a policy ecology of implementation of evidence-based practices in public mental health settings , 2008, Implementation science : IS.

[128]  Marije Bosch,et al.  Selecting, Tailoring, and implementing knowledge translation interventions , 2009 .

[129]  Christopher Gillberg,et al.  The Autism - Tics, AD/HD and other Comorbidities inventory (A-TAC): further validation of a telephone interview for epidemiological research , 2010, BMC psychiatry.

[130]  D. Sloan,et al.  Treating Trauma and Traumatic Grief in Children and Adolescents , 2007, Journal of Contemporary Psychotherapy.

[131]  J. Kirchner,et al.  Expert recommendations for implementing change (ERIC): protocol for a mixed methods study , 2014, Implementation Science.

[132]  Anthony L. Hemmelgarn,et al.  The Organizational Social Context of Mental Health Medicaid Waiver Programs with Family Support Services: Implications for Research and Practice , 2013, Administration and Policy in Mental Health and Mental Health Services Research.

[133]  C. McNeil,et al.  Evaluation of a Treatment Manual and Workshops for Disseminating, Parent–Child Interaction Therapy , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[134]  D. Sackett,et al.  Evidence based medicine: what it is and what it isn't , 1996, BMJ.

[135]  B. Mittman,et al.  An organizational framework and strategic implementation for system-level change to enhance research-based practice: QUERI Series , 2008, Implementation science : IS.

[136]  K. Hoagwood,et al.  Introduction to the Special Section on Practice Contexts: A Glimpse into the Nether World of Public Mental Health Services for Children and Families , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[137]  H. Soydan,et al.  Evidence-Based Clearinghouses in Social Work , 2010 .

[138]  E. Larson A tool to assess barriers to adherence to hand hygiene guideline. , 2004, American journal of infection control.

[139]  W. Ling,et al.  An intensive outpatient approach for cocaine abuse treatment. The Matrix model. , 1995, Journal of substance abuse treatment.

[140]  R. Raghavan,et al.  A preliminary analysis of the receipt of mental health services consistent with national standards among children in the child welfare system. , 2010, American journal of public health.

[141]  Bethany R. Lee,et al.  Translating the Common Elements Approach: Social Work's Experiences in Education, Practice, and Research , 2014, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[142]  M. Small `How many cases do I need?' , 2009 .

[143]  J. Weisz,et al.  Evidence-based youth psychotherapies versus usual clinical care: a meta-analysis of direct comparisons. , 2006, The American psychologist.

[144]  S. Mccracken,et al.  Practitioner Expertise in Evidence-Based Practice Decision Making , 2008 .

[145]  R. Yin,et al.  Case Study Research: Design and Methods (4th ed. , 2009 .

[146]  K. Hawley,et al.  Mental health clinicians' motivation to invest in training: results from a practice-based research network survey. , 2013, Psychiatric services.

[147]  Grant S. Fletcher,et al.  Strategies to implement alcohol screening and brief intervention in primary care settings: a structured literature review. , 2011, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[148]  Susan Michie,et al.  Specifying and reporting complex behaviour change interventions: the need for a scientific method , 2009, Implementation science : IS.

[149]  Anthony L. Hemmelgarn,et al.  Randomized trial of the Availability, Responsiveness and Continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. , 2013, Journal of the American Academy of Child and Adolescent Psychiatry.

[150]  M Davies,et al.  The NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC-2.3): description, acceptability, prevalence rates, and performance in the MECA Study. Methods for the Epidemiology of Child and Adolescent Mental Disorders Study. , 1996, Journal of the American Academy of Child and Adolescent Psychiatry.

[151]  C. Clancy,et al.  Guidelines for guidelines: we've come a long way. , 2007, Chest.

[152]  David A Chambers,et al.  Bridging research and practice: models for dissemination and implementation research. , 2012, American journal of preventive medicine.

[153]  J. Durlak,et al.  Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation , 2008, American journal of community psychology.

[154]  Stacy L. Frazier,et al.  Toward the Effective and Efficient Measurement of Implementation Fidelity , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[155]  E. Proctor,et al.  From Knowledge Production to Implementation: Research Challenges and Imperatives , 2008, Research on social work practice.

[156]  Duke Benadom Forward , 1996, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[157]  G. Aarons Mental Health Provider Attitudes Toward Adoption of Evidence-Based Practice: The Evidence-Based Practice Attitude Scale (EBPAS) , 2004, Mental health services research.

[158]  John Landsverk,et al.  The Cost of Implementing New Strategies (COINS): A Method for Mapping Implementation Resources Using the Stages of Implementation Completion. , 2014, Children and youth services review.

[159]  A. Edmondson Psychological Safety and Learning Behavior in Work Teams , 1999 .

[160]  A national survey of practicing psychologists' attitudes toward psychotherapy treatment manuals. , 2000, Journal of consulting and clinical psychology.

[161]  S. Michie,et al.  Validation of the theoretical domains framework for use in behaviour change and implementation research , 2012, Implementation Science.

[162]  Enola K. Proctor,et al.  A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health , 2012, Medical care research and review : MCRR.

[163]  Spencer G. Nabors,et al.  Clinical decisionmaking: opening the black box of cognitive reasoning. , 2006, Annals of emergency medicine.

[164]  Luke Vale,et al.  Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966-1998. , 2006, Journal of general internal medicine.

[165]  B. Powell A Mixed Methods Multiple Case Study of Implementation as Usual in Children's Social Service Organizations , 2014 .

[166]  K. Hofman,et al.  Implementation Science , 2007, Science.

[167]  Kimberley R. Isett,et al.  The state policy context of implementation issues for evidence-based practices in mental health. , 2007, Psychiatric services.

[168]  David A. Chambers,et al.  Developing Robust, Sustainable, Implementation Systems Using Rigorous, Rapid and Relevant Science , 2012, Clinical and translational science.

[169]  F. Kohlbacher The Use of Qualitative Content Analysis in Case Study Research , 2006 .

[170]  J. Lochman,et al.  Dissemination of the Coping Power program: importance of intensity of counselor training. , 2009, Journal of consulting and clinical psychology.

[171]  E. Rogers Diffusion of Innovations , 1962 .

[172]  Jacqueline A Pugh,et al.  Audit and feedback and clinical practice guideline adherence: Making feedback actionable , 2006, Implementation science : IS.

[173]  G. Aarons,et al.  Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. , 2012, Journal of the American Academy of Child and Adolescent Psychiatry.

[174]  S J Rolnick,et al.  Lessons from experienced guideline implementers: attend to many factors and use multiple strategies. , 2000, The Joint Commission journal on quality improvement.

[175]  R. Beidas,et al.  Consultation as an Implementation Strategy for Evidence-Based Practices Across Multiple Contexts: Unpacking the Black Box , 2013, Administration and Policy in Mental Health and Mental Health Services Research.

[176]  Richard Baker,et al.  A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice , 2013, Implementation Science.

[177]  L. Palinkas,et al.  Implementation of Evidence-based Practice in Child Welfare: Service Provider Perspectives , 2007, Administration and Policy in Mental Health and Mental Health Services Research.

[178]  K. Kelleher,et al.  A Survey of the Infrastructure for Children’s Mental Health Services: Implications for the Implementation of Empirically Supported Treatments (ESTs) , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[179]  Charles A. Rapp,et al.  Strategies for Improving Fidelity in the National Evidence-Based Practices Project , 2009 .

[180]  J. Unützer,et al.  Improving practice in community-based settings: a randomized trial of supervision – study protocol , 2013, Implementation Science.

[181]  Peter Mendel,et al.  Implementation of the CALM intervention for anxiety disorders: a qualitative study , 2012, Implementation Science.

[182]  L. Baker,et al.  The Role of Organizational Infrastructure in Implementation of Hospitals' Quality Improvement , 2006, Hospital topics.

[183]  P. Bebbington,et al.  Successful engagement: a mixed methods study of the approaches of assertive community treatment and community mental health teams in the REACT trial , 2009, Social Psychiatry and Psychiatric Epidemiology.

[184]  Gregory A Aarons,et al.  Organizational Culture and Climate and Mental Health Provider Attitudes Toward Evidence-Based Practice. , 2006, Psychological services.

[185]  Tom Lane,et al.  Consumers and families as partners in implementing evidence-based practice. , 2003, The Psychiatric clinics of North America.

[186]  J. Grimshaw,et al.  Systematic review of economic evaluations and cost analyses of guideline implementation strategies , 2007, The European Journal of Health Economics.

[187]  Pat Croskerry,et al.  The theory and practice of clinical decision-making , 2005 .

[188]  Alicia C. Bunger,et al.  Context matters: measuring implementation climate among individuals and groups , 2014, Implementation Science.

[189]  J. Cosby,et al.  Resilience Engineering: Concepts and Precepts , 2006, Quality and Safety in Health Care.

[190]  Brian S. Mittman,et al.  Implementation Science in Health Care , 2012 .

[191]  R. Pascale,et al.  The Power of Positive Deviance: How Unlikely Innovators Solve the World's Toughest Problems , 2010 .

[192]  K. Kelleher,et al.  Therapist Turnover and New Program Sustainability in Mental Health Clinics as a Function of Organizational Culture, Climate, and Service Structure , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[193]  G. Aarons,et al.  Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[194]  J. Lowery,et al.  Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science , 2009, Implementation science : IS.

[195]  Lawrence Palinkas,et al.  Design Elements in Implementation Research: A Structured Review of Child Welfare and Child Mental Health Studies , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[196]  Mary F Brunette,et al.  Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. , 2008, Psychiatric services.

[197]  Lawrence A. Palinkas,et al.  Mixed Method Designs in Implementation Research , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[198]  C. Abraham,et al.  Making psychological theory useful for implementing evidence based practice: a consensus approach , 2005, Quality and Safety in Health Care.

[199]  Mark Olfson,et al.  Service utilization for lifetime mental disorders in U.S. adolescents: results of the National Comorbidity Survey-Adolescent Supplement (NCS-A). , 2011, Journal of the American Academy of Child and Adolescent Psychiatry.

[200]  Andrew D Oxman,et al.  Closing the gap between research and practice : an overview of systematic reviews of interventions to promote the implementation of research findings , 2011 .

[201]  P. Schnurr,et al.  Measurement of a model of implementation for health care: toward a testable theory , 2012, Implementation Science.

[202]  ศิวะพร ภู่พันธ์,et al.  DESIGNING AND CONDUCTING MIXED METHODS RESEARCH (2nd Edition) , 2015 .

[203]  Rachel Haine-Schlagel,et al.  Mental health care for children with disruptive behavior problems: a view inside therapists' offices. , 2010, Psychiatric services.

[204]  R. Beidas,et al.  Training as Usual: Can Therapist Behavior Change After Reading a Manual and Attending a Brief Workship on Cognitive Behavioral Therapy for Youth Anxiety? , 2009 .

[205]  Jo Rycroft-Malone,et al.  An implementation research agenda , 2009, Implementation science : IS.

[206]  Anthony L. Hemmelgarn,et al.  Randomized trial of the Availability, Responsiveness, and Continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. , 2012, Journal of the American Academy of Child and Adolescent Psychiatry.

[207]  R. Whitley,et al.  Evidence-based Practice Implementation Strategies: Results of a Qualitative Study , 2008, Community Mental Health Journal.

[208]  N. Denzin,et al.  The discipline and practice of qualitative research [Introduction]. Denzin & Y. S. Lincoln (Eds.), The handbook of qualitative research (2nd ed. , 2000 .

[209]  F Davidoff,et al.  Toward stronger evidence on quality improvement. Draft publication guidelines: the beginning of a consensus project , 2005, Quality and Safety in Health Care.

[210]  J. Morse Qualitative data analysis (2nd ed): Mathew B. Miles and A. Michael Huberman. Thousand Oaks, CA: Sage Publications, 1994. Price: $65.00 hardback, $32.00 paperback. 238 pp , 1996 .

[211]  M. A. Scheirer Linking sustainability research to intervention types. , 2013, American journal of public health.

[212]  Ross C. Brownson,et al.  Dissemination and implementation research in health : translating science to practice , 2012 .

[213]  R. Beidas,et al.  Free, brief, and validated: Standardized instruments for low-resource mental health settings. , 2015, Cognitive and behavioral practice.

[214]  Enola K Proctor,et al.  A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions , 2014, Research on social work practice.

[215]  Mark W. Fraser,et al.  Intervention Research: Developing Social Programs , 2009 .

[216]  W. Neuman,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2002 .

[217]  Christopher A. Voss,et al.  Case research in operations management , 2002 .

[218]  Harlan M Krumholz,et al.  Research in action: using positive deviance to improve quality of health care , 2009, Implementation science : IS.

[219]  Karen A. Blase,et al.  Implementation Research: A Synthesis of the , 2005 .

[220]  Lawrence R. James,et al.  The cross‐level effects of culture and climate in human service teams , 2002 .

[221]  R. Yin Case Study Research: Design and Methods , 1984 .

[222]  A. Carson-Stevens,et al.  Leading change in health-care quality with the Institute for Healthcare Improvement Open School. , 2012, British journal of hospital medicine.

[223]  J. Grimshaw,et al.  Evidence-based implementation of evidence-based medicine. , 1999, The Joint Commission journal on quality improvement.

[224]  S. Flottorp,et al.  Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. , 2005, The Cochrane database of systematic reviews.

[225]  H. Bernard Research Methods in Anthropology: Qualitative and Quantitative Approaches , 1988 .

[226]  R. Barth,et al.  Common components of parenting programs for children birth to eight years of age involved with child welfare services , 2014 .

[227]  K. Seers Qualitative data analysis , 2011, Evidence Based Nursing.

[228]  S. Straus,et al.  Knowledge translation in health care : moving from evidence to practice , 2009 .

[229]  R. Brownson,et al.  Barriers to Evidence-Based Decision Making in Public Health: A National Survey of Chronic Disease Practitioners , 2010, Public health reports.

[230]  F Davidoff,et al.  Publication guidelines for quality improvement in health care: evolution of the SQUIRE project , 2008, Quality & Safety in Health Care.

[231]  T. Valente Network Interventions , 2012, Science.

[232]  M. Eccles,et al.  Welcome to Implementation Science , 2006, Implementation Science.

[233]  S. Marcus,et al.  Training and consultation to promote implementation of an empirically supported treatment: a randomized trial. , 2012, Psychiatric services.