Use of Human-Centered Design to Improve Implementation of Evidence-Based Psychotherapies in Low-Resource Communities: Protocol for Studies Applying a Framework to Assess Usability


Background This paper presents the protocol for the National Institute of Mental Health (NIMH)–funded University of Washington’s ALACRITY (Advanced Laboratories for Accelerating the Reach and Impact of Treatments for Youth and Adults with Mental Illness) Center (UWAC), which uses human-centered design (HCD) methods to improve the implementation of evidence-based psychosocial interventions (EBPIs). We propose that usability—the degree to which interventions and implementation strategies can be used with ease, efficiency, effectiveness, and satisfaction—is a fundamental, yet poorly understood determinant of implementation. Objective We present a novel Discover, Design/Build, and Test (DDBT) framework to study usability as an implementation determinant. DDBT will be applied across Center projects to develop scalable and efficient implementation strategies (eg, training tools), modify existing EBPIs to enhance usability, and create usable and nonburdensome decision support tools for quality delivery of EBPIs. Methods Stakeholder participants will be implementation practitioners/intermediaries, mental health clinicians, and patients with mental illness in nonspecialty mental health settings in underresourced communities. Three preplanned projects and 12 pilot studies will employ the DDBT model to (1) identify usability challenges in implementing EBPIs in underresourced settings; (2) iteratively design solutions to overcome these challenges; and (3) compare the solution to the original version of the EPBI or implementation strategy on usability, quality of care, and patient-reported outcomes. The final products from the center will be a streamlined modification and redesign model that will improve the usability of EBPIs and implementation strategies (eg, tools to support EBPI education and decision making); a matrix of modification targets (ie, usability issues) that are both common and unique to EBPIs, strategies, settings, and patient populations; and a compilation of redesign strategies and the relative effectiveness of the redesigned solution compared to the original EBPI or strategy. Results The UWAC received institutional review board approval for the three separate studies in March 2018 and was funded in May 2018. Conclusions The outcomes from this center will inform the implementation of EBPIs by identifying cross-cutting features of EBPIs and implementation strategies that influence the use and acceptability of these interventions, actively involving stakeholder clinicians and implementation practitioners in the design of the EBPI modification or implementation strategy solution and identifying the impact of HCD-informed modifications and solutions on intervention effectiveness and quality. Trial Registration ClinicalTrials.gov NCT03515226 (https://clinicaltrials.gov/ct2/show/NCT03515226), NCT03514394 (https://clinicaltrials.gov/ct2/show/NCT03514394), and NCT03516513 (https://clinicaltrials.gov/ct2/show/NCT03516513). International Registered Report Identifier (IRRID) DERR1-10.2196/14990

[1]  Christopher J. Miller,et al.  The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions , 2019, Implementation Science.

[2]  Jennifer Leeman,et al.  Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice , 2017, Implementation Science.

[3]  H. Goldman,et al.  Public policy and evidence-based practice. , 2003, The Psychiatric clinics of North America.

[4]  J. Kirchner,et al.  A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project , 2015, Implementation Science.

[5]  Clayton Lewis,et al.  Designing for usability—key principles and what designers think , 1983, CHI '83.

[6]  Lawrence A. Palinkas,et al.  Mixed Method Designs in Implementation Research , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[7]  Cara C. Lewis,et al.  Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria , 2015, Implementation Science.

[8]  M. Olfson,et al.  Trends in office-based mental health care provided by psychiatrists and primary care physicians. , 2014, The Journal of clinical psychiatry.

[9]  Nicholas Ladany,et al.  Consensual Qualitative Research: An Update , 2005 .

[10]  Gavriel Salvendy,et al.  The cognitive task analysis methods for job and task design: review and reappraisal , 2004, Behav. Inf. Technol..

[11]  Anna-Lisa Osvalder,et al.  Enhanced Cognitive Walkthrough: Development of the Cognitive Walkthrough Method to Better Predict, Identify, and Present Usability Problems , 2013, Adv. Hum. Comput. Interact..

[12]  Enola K Proctor,et al.  Implementation strategies: recommendations for specifying and reporting , 2013, Implementation Science.

[13]  Cara C. Lewis,et al.  From Classification to Causality: Advancing Understanding of Mechanisms of Change in Implementation Science , 2018, Front. Public Health.

[14]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[15]  Gavriel Salvendy,et al.  Number of people required for usability evaluation , 2010, Commun. ACM.

[16]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[17]  R. Rapee,et al.  A Model of Therapist Competencies for the Empirically Supported Cognitive Behavioral Treatment of Child and Adolescent Anxiety and Depressive Disorders , 2011, Clinical child and family psychology review.

[18]  M. Bauer,et al.  Effectiveness-implementation hybrid designs: implications for quality improvement science , 2013, Implementation Science.

[19]  Aaron R. Lyon,et al.  User-Centered Design for Psychosocial Intervention Development and Implementation. , 2016, Clinical psychology : a publication of the Division of Clinical Psychology of the American Psychological Association.

[20]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[21]  Aaron R Lyon,et al.  Accelerating Digital Mental Health Research From Early Design and Creation to Successful Implementation and Sustainment , 2017, Journal of medical Internet research.

[22]  Mark Olfson,et al.  Service utilization for lifetime mental disorders in U.S. adolescents: results of the National Comorbidity Survey-Adolescent Supplement (NCS-A). , 2011, Journal of the American Academy of Child and Adolescent Psychiatry.

[23]  Arthur C. Graesser,et al.  Advances from the Office of Naval Research STEM Grand Challenge: expanding the boundaries of intelligent tutoring systems , 2018, International journal of STEM education.

[24]  Martti Mäntylä,et al.  How Effective Are User Studies? , 2000, BCS HCI.

[25]  R. Spitzer,et al.  The PHQ-9: validity of a brief depression severity measure. , 2001, Journal of general internal medicine.

[26]  Patricia L. Kohl,et al.  Intermediary/purveyor organizations for evidence-based interventions in the US child mental health: characteristics and implementation strategies , 2019, Implementation Science.

[27]  R. Kravitz,et al.  Depression Treatment Preferences of Hispanic Individuals: Exploring the Influence of Ethnicity, Language, and Explanatory Models , 2011, The Journal of the American Board of Family Medicine.

[28]  P. Areán,et al.  Comparing engage with PST in late-life major depression: a preliminary report. , 2015, The American journal of geriatric psychiatry : official journal of the American Association for Geriatric Psychiatry.

[29]  G. Aarons,et al.  Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[30]  Danielle L. Fettes,et al.  Collaboration, Negotiation, and Coalescence for Interagency-Collaborative Teams to Scale-Up Evidence-Based Practice , 2014, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[31]  P. Resick,et al.  Cognitive processing therapy for sexual assault victims. , 1992, Journal of consulting and clinical psychology.

[32]  Telephone-based cognitive-behavioral therapy for Latino patients living in rural areas: a randomized pilot study. , 2011, Psychiatric services.

[33]  David A Chambers,et al.  The Adaptome: Advancing the Science of Intervention Adaptation. , 2016, American journal of preventive medicine.

[34]  C. Hill,et al.  A Guide to Conducting Consensual Qualitative Research , 1997 .

[35]  Adrian Angold,et al.  Pathways into and through mental health services for children and adolescents. , 2003, Psychiatric services.

[36]  Alicia C. Bunger,et al.  Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[37]  Jonathan Grudin,et al.  Personas, Participatory Design and Product Development: An Infrastructure for Engagement , 2002 .

[38]  Andrew C. Leon,et al.  Assessing Psychiatric Impairment in Primary Care with the Sheehan Disability Scale , 1997, International journal of psychiatry in medicine.

[39]  J. Hay,et al.  Depression treatment preferences of older white and Mexican origin men. , 2013, General hospital psychiatry.

[40]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[41]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[42]  E. Ludman,et al.  Telephone-based cognitive-behavioral therapy for Latino patients living in rural areas: a randomized pilot study. , 2011, Psychiatric services.

[43]  L. G. Tornatzky,et al.  Innovation characteristics and innovation adoption-implementation: A meta-analysis of findings , 1982, IEEE Transactions on Engineering Management.

[44]  Nathaniel J. Williams Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice , 2016, Administration and Policy in Mental Health and Mental Health Services Research.

[45]  Aaron R Lyon,et al.  User-Centered Redesign of Evidence-Based Psychosocial Interventions to Enhance Implementation-Hospitable Soil or Better Seeds? , 2019, JAMA psychiatry.

[46]  M. Lindén,et al.  Challenges in the Implementation of Manualized Psychotherapy in Combat-related PTSD , 2008, CNS Spectrums.

[47]  Martin C. Maguire,et al.  Methods to support human-centred design , 2001, Int. J. Hum. Comput. Stud..

[48]  G. Sedlar,et al.  Clinician-Reported Modification to a CBT Approach in Children’s Mental Health , 2020, Journal of Emotional and Behavioral Disorders.

[49]  Alex R Dopp,et al.  Integrating implementation and user-centred design strategies to enhance the impact of health services: protocol from a concept mapping study , 2019, Health Research Policy and Systems.

[50]  Steven M. Belz,et al.  The user action framework: a reliable foundation for usability engineering support tools , 2001, Int. J. Hum. Comput. Stud..

[51]  M. Hegel,et al.  Sustained Adoption of an Evidence-Based Treatment: A Survey of Clinicians Certified in Problem-Solving Therapy , 2012, Depression research and treatment.

[52]  Caitlin N. Dorsey,et al.  Psychometric assessment of three newly developed implementation outcome measures , 2017, Implementation Science.

[53]  F. Lespérance,et al.  Treatment preferences in patients with first episode depression. , 2013, Journal of affective disorders.

[54]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[55]  A. Nezu Efficacy of a social problem-solving therapy approach for unipolar depression. , 1986, Journal of consulting and clinical psychology.

[56]  D. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[57]  B. Meyers,et al.  Preferences for depression treatment among elderly home health care patients. , 2011, Psychiatric Services.

[58]  Aaron R. Lyon,et al.  Toward high‐fidelity treatment as usual: Evidence‐based intervention structures to improve usual care psychotherapy , 2018, Clinical Psychology: Science and Practice.

[59]  Geoffrey M. Curran,et al.  Effectiveness-implementation Hybrid Designs: Combining Elements of Clinical Effectiveness and Implementation Research to Enhance Public Health Impact , 2012, Medical care.

[60]  David A. Chambers,et al.  Advancing the Science of Implementation: A Workshop Summary , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[61]  D. Sheehan,et al.  The measurement of disability. , 1996 .

[62]  R. Raghavan The role of economic evaluation in dissemination and implementation research , 2012 .

[63]  R. Khajouei,et al.  Classification and prioritization of usability problems using an augmented classification scheme , 2011, J. Biomed. Informatics.

[64]  Kathryn E Parisi,et al.  A glossary of user-centered design strategies for implementation experts. , 2018, Translational behavioral medicine.

[65]  K. Dobson,et al.  The Cognitive Therapy Scale: psychometric properties. , 1986, Journal of consulting and clinical psychology.

[66]  Diane R. Gehart The core competencies and MFT education: practical aspects of transitioning to a learning-centered, outcome-based pedagogy. , 2011, Journal of marital and family therapy.

[67]  Karen Holtzblatt,et al.  Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design , 2004, UBIQ.

[68]  S. Thielke,et al.  Racial and ethnic differences in receipt of antidepressants and psychotherapy by veterans with chronic depression. , 2014, Psychiatric services.

[69]  D. Sheehan,et al.  The measurement of disability , 1996, International clinical psychopharmacology.

[70]  R. Franks,et al.  Who Supports the Successful Implementation and Sustainability of Evidence-Based Practices? Defining and Understanding the Roles of Intermediary and Purveyor Organizations. , 2015, New directions for child and adolescent development.

[71]  Sean A. Munson,et al.  Translational Resources: Reducing the Gap Between Academic Research and HCI Practice , 2017, Conference on Designing Interactive Systems.

[72]  Alan Cooper,et al.  About Face 3: the essentials of interaction design , 1995 .

[73]  Richard Baker,et al.  A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice , 2013, Implementation Science.

[74]  James Wilson,et al.  Rapid Prototyping for User Interface Design , 1988 .

[75]  Riitta Jääskeläinen Think-aloud protocol , 2010 .

[76]  Aaron R. Lyon,et al.  Closing the Science–Practice Gap in Implementation Before It Widens , 2020 .

[77]  Bruce F Chorpita,et al.  Understanding the common elements of evidence-based practice: misconceptions and clinical examples. , 2007, Journal of the American Academy of Child and Adolescent Psychiatry.

[78]  M. Otto,et al.  Patient preference for psychological vs pharmacologic treatment of psychiatric disorders: a meta-analytic review. , 2013, The Journal of clinical psychiatry.

[79]  D. Chambless,et al.  Decision Making and the Use of Evidence-Based Practice: Is the Three-Legged Stool Balanced? , 2018, Practice innovations.

[80]  P. Areán,et al.  Implementing evidence-based psychotherapies in settings serving older adults: challenges and solutions. , 2012, Psychiatric services.

[81]  Byron J. Powell,et al.  Adapting a Compilation of Implementation Strategies to Advance School-Based Implementation Research and Practice , 2019, Prevention Science.

[82]  P. Kendall,et al.  Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. , 2010, Clinical psychology : a publication of the Division of Clinical Psychology of the American Psychological Association.

[83]  Rinad S. Beidas,et al.  Methods to Improve the Selection and Tailoring of Implementation Strategies , 2017, The Journal of Behavioral Health Services & Research.

[84]  J. Lowery,et al.  Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science , 2009, Implementation science : IS.

[85]  P. Areán,et al.  "Engage" Therapy: Behavioral Activation and Improvement of Late-Life Major Depression. , 2016, The American journal of geriatric psychiatry : official journal of the American Association for Geriatric Psychiatry.

[86]  Marjo Kauppinen,et al.  Identifying and selecting users for user-centered design , 2004, NordiCHI '04.

[87]  L. Faulkner Beyond the five-user assumption: Benefits of increased sample sizes in usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[88]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[89]  Cathleen Willging,et al.  Interagency Collaborative Team Model for Capacity Building to Scale-Up Evidence-Based Practice. , 2014, Children and youth services review.

[90]  Aaron R. Lyon,et al.  Using Modular Psychotherapy in School Mental Health: Provider Perspectives on Intervention-Setting Fit , 2014, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.