Alpha test results for a Housing First eLearning strategy: the value of multiple qualitative methods for intervention design

BackgroundDetailed descriptions of implementation strategies are lacking, and there is a corresponding dearth of information regarding methods employed in implementation strategy development. This paper describes methods and findings related to the alpha testing of eLearning modules developed as part of the Housing First Technical Assistance and Training (HFTAT) program’s development. Alpha testing is an approach for improving the quality of a product prior to beta (i.e., real world) testing with potential applications for intervention development.MethodsTen participants in two cities tested the modules. We collected data through (1) a structured log where participants were asked to record their experiences as they worked through the modules; (2) a brief online questionnaire delivered at the end of each module; and (3) focus groups.ResultsThe alpha test provided useful data related to the acceptability and feasibility of eLearning as an implementation strategy, as well as identifying a number of technical issues and bugs. Each of the qualitative methods used provided unique and valuable information. In particular, logs were the most useful for identifying technical issues, and focus groups provided high quality data regarding how the intervention could best be used as an implementation strategy.ConclusionsAlpha testing was a valuable step in intervention development, providing us an understanding of issues that would have been more difficult to address at a later stage of the study. As a result, we were able to improve the modules prior to pilot testing of the entire HFTAT. Researchers wishing to alpha test interventions prior to piloting should balance the unique benefits of different data collection approaches with the need to minimize burdens for themselves and participants.

[1]  Incongruence Between Trauma Center Social Workers’ Beliefs About Substance Use Interventions and Intentions to Intervene , 2015, Social work in health care.

[2]  Lisa Saldana,et al.  Supporting Implementation: The Role of Community Development Teams to Build Infrastructure , 2012, American journal of community psychology.

[3]  Razvan Rughinis.  Talkative objects in need of interpretation. re-thinking digital badges in education , 2013, CHI Extended Abstracts.

[4]  A. Stoddard,et al.  Online self-administered training for post-traumatic stress disorder treatment providers: design and methods for a randomized, prospective intervention study , 2012, Implementation Science.

[5]  Sally Redman,et al.  The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. , 2013, Health promotion international.

[6]  Anne E. Figert,et al.  Connecting fractured lives to a fragmented system: Chicago housing for health partnership , 2008 .

[7]  Timo Kinnunen,et al.  Applying user-centered design to mobile application development , 2005, CACM.

[8]  Hsiu-Fang Hsieh,et al.  Three Approaches to Qualitative Content Analysis , 2005, Qualitative health research.

[9]  Terry Anderson,et al.  Towards a Theory of Online Learning , 2004 .

[10]  Lucy Yardley,et al.  The person-based approach to enhancing the acceptability and feasibility of interventions , 2015, Pilot and Feasibility Studies.

[11]  M. Patton.  Qualitative research and evaluation methods , 1980 .

[12]  Ivana Acocella.  The focus groups in social research: advantages and disadvantages , 2012 .

[13]  Titus K L Schleyer,et al.  Developing high-quality educational software. , 2003, Journal of dental education.

[14]  P. Harris,et al.  Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support , 2009, J. Biomed. Informatics.

[15]  P. Hoddinott.  A new era for intervention development studies , 2015, Pilot and Feasibility Studies.

[16]  M. Clark.  Narrative learning: Its contours and its possibilities , 2010 .

[17]  Francisco Pablo Holgado Tello,et al.  Training Satisfaction Rating Scale Development of a Measurement Model Using Polychoric Correlations , 2006 .

[18]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[19]  Graham Moore,et al.  Contextual issues and qualitative research , 2016 .

[20]  Penny Garcia,et al.  Digital Storytelling: A New Player on the Narrative Field , 2010 .

[21]  A. Huberman,et al.  Qualitative Data Analysis: A Methods Sourcebook , 1994 .

[22]  D. Blevins,et al.  Translating clinical training into practice in complex mental health systems: Toward opening the 'Black Box' of implementation , 2008, Implementation science : IS.

[23]  D. Watson,et al.  Understanding the Critical Ingredients for Facilitating Consumer Change in Housing First Programming: A Case Study Approach , 2013, The Journal of Behavioral Health Services & Research.

[24]  David A. Chambers,et al.  Implementation Research in Mental Health Services: an Emerging Science with Conceptual, Methodological, and Training challenges , 2008, Administration and Policy in Mental Health and Mental Health Services Research.

[25]  Paul Hazel.  Toward a narrative pedagogy for interactive learning environments , 2008, Interact. Learn. Environ..

[26]  Huiping Xu,et al.  Development and testing of an implementation strategy for a complex housing intervention: protocol for a mixed methods study , 2014, Implementation Science.

[27]  J. Kirchner,et al.  A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project , 2015, Implementation Science.

[28]  Kim Flintoff,et al.  Digital badges in education , 2013, Education and Information Technologies.

[29]  Anders Bruun,et al.  Let your users do the testing: a comparison of three remote asynchronous usability testing methods , 2009, CHI.

[30]  R. Mayer.  Applying the science of learning to medical education , 2010, Medical education.

[31]  Wei Li,et al.  Cultural influences on knowledge sharing through online communities of practice , 2006, J. Knowl. Manag..

[32]  J. Kirchner,et al.  Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study , 2015, Implementation Science.

[33]  H. Rosenberg,et al.  Acceptability and availability of harm-reduction interventions for drug abuse in American substance abuse treatment agencies. , 2003, Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors.

[34]  Linda S. Lotto.  Qualitative Data Analysis: A Sourcebook of New Methods , 1986 .

[35]  John Orwat,et al.  The housing first model (HFM) fidelity index: designing and testing a tool for measuring integrity of housing programs that serve active substance users , 2013, Substance Abuse Treatment, Prevention, and Policy.

[36]  M. Olson,et al.  Enhancing Adult Motivation to Learn: A Comprehensive Guide for Teaching All Adults , 2010 .

[37]  S. Tsemberis,et al.  Housing stability and recovery among chronically homeless persons with co-occuring disorders in Washington, DC. , 2012, American journal of public health.

[38]  G A Marlatt,et al.  Harm reduction: come as you are. , 1996, Addictive behaviors.

[39]  Alicia C. Bunger,et al.  Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda , 2010, Administration and Policy in Mental Health and Mental Health Services Research.

[40]  P. Kendall,et al.  Dissemination and Implementation of Evidence-Based Practices: Training and Consultation as Implementation Strategies. , 2013, Clinical psychology : a publication of the Division of Clinical Psychology of the American Psychological Association.

[41]  G. McHugo,et al.  Evidence-Based Practice Implementation in Community Mental Health Settings: The Relative Importance of Key Domains of Implementation Activity , 2012, Administration and Policy in Mental Health and Mental Health Services Research.

[42]  M. van Dijk,et al.  Do implementation strategies increase adherence to pain assessment in hospitals? A systematic review. , 2013, International journal of nursing studies.

[43]  T. Reay,et al.  Organizational impact of evidence-informed decision making training initiatives: a case study comparison of two approaches , 2014, Implementation Science.

[44]  Vladan Devedzic,et al.  Open Badges: Challenges and Opportunities , 2014, ICWL.