A Narrative Review of Methods for Applying User Experience in the Design and Assessment of Mental Health Smartphone Interventions

Abstract Objectives User experience (UX) plays a key role in uptake and usage of mental health smartphone interventions, yet remains underinvestigated. This review aimed to characterize and compare UX evaluation approaches that have been applied in this specific context, and to identify implications for research and practice. Methods A narrative review was conducted of UX-themed studies published in PubMed, PsycINFO, and Scopus up to February 2019. Eligible studies reported on data reflecting users' interactions with a smartphone intervention for any mental health condition. Studies were categorized into “situated” versus “construct-based” methods according to whether or not an established UX construct was used to acquire and analyze data. Results Situated approaches used bespoke UX metrics, including quantitative measures of usage and performance, as well as grounded interview data. Construct-based approaches such as assessments of usability and acceptability were based on conceptual frameworks, with methodologically stronger versions featuring construct definitions, validated measurement tools, and an ability to compare data across studies. Constructs and measures were sometimes combined to form bespoke construct-based approaches. Conclusions Both situated and construct-based UX data may provide benefits during design and implementation of a mental health smartphone intervention by helping to clarify the needs of users and the impact of new features. Notable however was the omission of UX methods, such as split testing. Future research should consider these unaddressed methods, aim to improve the rigor of UX assessment, integrate their use alongside clinical outcomes, and explore UX assessment of more complex, adaptive interventions.

[1]  Inocencio Maramba,et al.  Methods of usability testing in the development of eHealth applications: A scoping review , 2019, Int. J. Medical Informatics.

[2]  John Torous,et al.  Towards a consensus around standards for smartphone apps and digital mental health , 2019, World psychiatry : official journal of the World Psychiatric Association.

[3]  V. Lawrence,et al.  Barriers to accessing mental health services for women with perinatal mental illness: systematic review and meta-synthesis of qualitative studies in the UK , 2019, BMJ Open.

[4]  I. Granic,et al.  Use of the Principles of Design Thinking to Address Limitations of Digital Mental Health Interventions for Youth: Viewpoint , 2019, Journal of medical Internet research.

[5]  Usability Testing of a Mobile App to Report Medication Errors Anonymously: Mixed-Methods Approach , 2018, JMIR human factors.

[6]  Russell Fulmer,et al.  Using Psychological Artificial Intelligence (Tess) to Relieve Symptoms of Depression and Anxiety: Randomized Controlled Trial , 2018, JMIR mental health.

[7]  Adrian B. R. Shatte,et al.  A Mobile App–Based Intervention for Depression: End-User and Expert Usability Testing Study , 2018, JMIR mental health.

[8]  John Torous,et al.  Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements , 2018, Evidence Based Journals.

[9]  J. Torous,et al.  A Hierarchical Framework for Evaluation and Informed Decision Making Regarding Smartphone Apps for Clinical Care. , 2018, Psychiatric services.

[10]  J. Niehaus,et al.  Using a Mobile Application in the Treatment of Dysregulated Anger Among Veterans. , 2017, Military medicine.

[11]  John Torous,et al.  The WPA-Lancet Psychiatry Commission on the Future of Psychiatry. , 2017, The lancet. Psychiatry.

[12]  Michael L. Birnbaum,et al.  A Systematic Review and Taxonomy of Published Quality Criteria Related to the Evaluation of User-Facing eHealth Programs , 2017, Journal of Medical Systems.

[13]  C. Meyer,et al.  A systematic review of the psychometric properties, usability and clinical impacts of mobile mood-monitoring applications in young people , 2017, Psychological Medicine.

[14]  J. Larsen,et al.  Using a mobile health application to reduce alcohol consumption: a mixed-methods evaluation of the drinkaware track & calculate units application , 2017, BMC Public Health.

[15]  John Torous,et al.  Methodology and Reporting of Mobile Health and Smartphone Application Studies for Schizophrenia , 2017, Harvard review of psychiatry.

[16]  Helen Christensen,et al.  The Reviews Are in: A Qualitative Content Analysis of Consumer Perspectives on Apps for Bipolar Disorder , 2017, Journal of medical Internet research.

[17]  Susanne Iwarsson,et al.  Research with and about user participation: potentials and challenges , 2017, Aging Clinical and Experimental Research.

[18]  S. Knaak,et al.  Mental illness-related stigma in healthcare , 2017, Healthcare management forum.

[19]  M. McConnell,et al.  The Use of Smartphones for Health Research. , 2017, Academic medicine : journal of the Association of American Medical Colleges.

[20]  G. Thornicroft,et al.  Interventions to reduce discrimination and stigma: the state of the art , 2017, Social Psychiatry and Psychiatric Epidemiology.

[21]  Yogesan Kanagasingam,et al.  End-user acceptance of a cloud-based teledentistry system and Android phone app for remote screening for oral diseases , 2017, Journal of telemedicine and telecare.

[22]  Sruthy Orozhiyathumana Agnisarman,et al.  Lessons learned from the usability assessment of home-based telemedicine systems. , 2017, Applied ergonomics.

[23]  Elizabeth Cummings,et al.  Design Thinking for mHealth Application Co-Design to Support Heart Failure Self-Management , 2017, CSHI.

[24]  Ben Matthews,et al.  The promise and the reality: a mental health workforce perspective on technology-enhanced youth mental health service delivery , 2016, BMC Health Services Research.

[25]  Comparing applets and oranges: barriers to evidence-based practice for app-based psychological interventions , 2016, Evidence-Based Mental Health.

[26]  J. Koziol-Mclain,et al.  Evaluation Methods for Assessing Users’ Psychological Experiences of Web-Based Psychosocial Interventions: A Systematic Review , 2016, Journal of medical Internet research.

[27]  Andrew T. Campbell,et al.  Mobile Behavioral Sensing for Outpatients and Inpatients With Schizophrenia. , 2016, Psychiatric services.

[28]  Kiley Sobel,et al.  User Experience Evaluation of a Smoking Cessation App in People With Serious Mental Illness. , 2016, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[29]  N. Rickard,et al.  Mental Health Smartphone Apps: Review and Evidence-Based Recommendations for Future Developments , 2016, JMIR mental health.

[30]  A. Lowell,et al.  Acceptability of Mental Health Apps for Aboriginal and Torres Strait Islander Australians: A Qualitative Study , 2016, Journal of medical Internet research.

[31]  J. Naslund,et al.  Feasibility of popular m-health technologies for activity tracking among individuals with serious mental illness. , 2015, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[32]  Oksana Zelenko,et al.  Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps , 2015, JMIR mHealth and uHealth.

[33]  S. Sabesan,et al.  Implementing telehealth as core business in health services , 2015, The Medical journal of Australia.

[34]  J. L. Bender,et al.  Finding a Depression App: A Review and Content Analysis of the Depression App Marketplace , 2015, JMIR mHealth and uHealth.

[35]  L. Yardley,et al.  The Person-Based Approach to Intervention Development: Application to Digital Health-Related Behavior Change Interventions , 2015, Journal of medical Internet research.

[36]  D. Ben-Zeev,et al.  Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. , 2014, Schizophrenia bulletin.

[37]  Mohammad Iranmanesh,et al.  Determinants of Telemedicine Acceptance in Selected Public Hospitals in Malaysia: Clinical Perspective , 2014, Journal of Medical Systems.

[38]  Martin Gaedke,et al.  Ensuring Web Interface Quality through Usability-Based Split Testing , 2014, ICWE.

[39]  David Coyle,et al.  Qualitative Meta-Synthesis of User Experience of Computerised Therapy for Depression and Anxiety , 2014, PloS one.

[40]  Aga Bojko,et al.  Eye Tracking the User Experience: A Practical Guide to Research , 2013 .

[41]  Chris Howe,et al.  Card sort analysis best practices , 2013 .

[42]  Philip T. Kortum,et al.  Usability Ratings for Everyday Products Measured With the System Usability Scale , 2013, Int. J. Hum. Comput. Interact..

[43]  V. Shute,et al.  What Is Design Thinking and Why Is It Important? , 2012 .

[44]  B Sheehan,et al.  A comparison of usability factors of four mobile devices for accessing healthcare information by adolescents. , 2012, Applied clinical informatics.

[45]  H. Christensen,et al.  Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review , 2010, BMC psychiatry.

[46]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[47]  Virpi Roto,et al.  Understanding, scoping and defining user experience: a survey approach , 2009, CHI.

[48]  Andrew F Long,et al.  Acceptability and satisfaction with a telecarer approach to the management of type 2 diabetes. , 2005, Diabetes care.

[49]  Fred D. Davis,et al.  A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies , 2000, Management Science.

[50]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[51]  James R. Lewis Psychometric Evaluation of the Post-Study System Usability Questionnaire: The PSSUQ , 1992 .