Assessing usability of eHealth technology: A comparison of usability benchmarking instruments

BACKGROUND It is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients' health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context. OBJECTIVES In this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use. METHODS A usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8). RESULTS The main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments. CONCLUSIONS With this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth.

[1]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[2]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[3]  E. L. Werner,et al.  Self-Management Skills in Chronic Disease Management , 2016, Medical decision making : an international journal of the Society for Medical Decision Making.

[4]  Jeff Sauro,et al.  Quantifying the User Experience: Practical Statistics for User Research , 2012 .

[5]  Ali Idri,et al.  A systematic review of gamification in e-Health , 2017, J. Biomed. Informatics.

[6]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[7]  E. B. Wilson.  Probable Inference, the Law of Succession, and Statistical Inference , 1927 .

[8]  Gunilla C. Nilsson,et al.  Health care consumers' experiences of information communication technology - A summary of literature , 2007, Int. J. Medical Informatics.

[9]  James R. Lewis,et al.  Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ , 1991, SGCH.

[10]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[11]  Amanda S. Newton,et al.  Usability, learnability and performance evaluation of Intelligent Research and Intervention Software: A delivery platform for eHealth interventions , 2016, Health Informatics J..

[12]  Thea van der Geest,et al.  Identifying Usability Issues for Personalization During Formative Evaluations: A Comparison of Three Methods , 2011, Int. J. Hum. Comput. Interact..

[13]  James R. Lewis,et al.  The System Usability Scale: Past, Present, and Future , 2018, Int. J. Hum. Comput. Interact..

[14]  Wiro J. Niessen,et al.  Older Age Relates to Worsening of Fine Motor Skills: A Population-Based Study of Middle-Aged and Elderly Persons , 2014, Front. Aging Neurosci..

[15]  Hermie Hermens,et al.  Usability in telemedicine systems - A literature survey , 2016, Int. J. Medical Informatics.

[16]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[17]  Claudia Ziegler Acemyan,et al.  How low can you go?: is the system usability scale range restricted? , 2013 .

[18]  Michael D. Byrne,et al.  Voting on a Smartphone , 2011 .

[19]  T. Shors.  Stressful experience and learning across the lifespan. , 2006, Annual review of psychology.

[20]  Rafael Rieder,et al.  Usability Evaluation Methods for Gesture-Based Games: A Systematic Review , 2016, JMIR serious games.

[21]  Panagiotis D. Bamidis,et al.  Moving Real Exergaming Engines on the Web: The webFitForAll Case Study in an Active and Healthy Ageing Living Lab Environment , 2017, IEEE Journal of Biomedical and Health Informatics.

[22]  B. McEwen.  Central effects of stress hormones in health and disease: Understanding the protective and damaging effects of stress and stress mediators. , 2008, European journal of pharmacology.

[23]  Simon McCallum,et al.  Gamification and Serious Games for Personalized Health , 2012, pHealth.

[24]  Abdelkamel Tari,et al.  Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis , 2016, Comput. Hum. Behav..

[25]  Nancy Staggers,et al.  Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics , 2015, J. Am. Medical Informatics Assoc..

[26]  J. H. Steiger.  Tests for comparing elements of a correlation matrix. , 1980 .

[27]  K. Rothman,et al.  Epidemiologic Analysis with a Programmable Calculator , 1982 .

[28]  Joel C. Perry,et al.  Improving patient motivation in game development for motor deficit rehabilitation , 2008, ACE '08.

[29]  Robert A Verheij,et al.  Benefits and problems of electronic information exchange as perceived by health care professionals: an interview study , 2011, BMC health services research.

[30]  Kathrin Maria Gerling,et al.  Game Design for Older Adults: Effects of Age-Related Changes on Structural Elements of Digital Games , 2012, ICEC.

[31]  Christos Katsanos,et al.  Perceived usability evaluation of learning management systems: Empirical evaluation of the System Usability Scale , 2015 .

[32]  Hannes Kaufmann,et al.  A Serious Exergame for Patients Suffering from Chronic Musculoskeletal Back and Neck Pain: A Pilot Study. , 2013, Games for health journal.

[33]  Philip T. Kortum,et al.  Usability Ratings for Everyday Products Measured With the System Usability Scale , 2013, Int. J. Hum. Comput. Interact..

[34]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[35]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.