A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support

Mobile health platforms offer significant opportunities for improving diabetic self-care, but only if adequate usability exists. Expert evaluations such as heuristic evaluation can provide distinct usability information about systems. The purpose of this study was to complete a usability evaluation of a mobile health system for diabetes patients using a modified heuristic evaluation technique of (1) dual-domain experts (healthcare professionals, usability experts), (2) validated scenarios and user tasks related to patients’ self-care, and (3) in-depth severity factor ratings. Experts identified 129 usability problems with 274 heuristic violations for the system. The categories Consistency and Standards dominated at 24.1% (n = 66), followed by Match Between System and Real World at 22.3% (n = 61). Average severity ratings across system views were 2.8 (of 4), with 9.3% (n = 12) rated as catastrophic and 53.5% (n = 69) as major. The large volume of violations with severe ratings indicated clear priorities for redesign. The modified heuristic approach allowed evaluators to identify unique and important issues, including ones related to self-management and patient safety. This article provides a template for one type of expert evaluation adding to the informaticists’ toolbox when needing to conduct a fast, resource-efficient and user-oriented heuristic evaluation.

[1]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[2]  Jakob Nielsen,et al.  Improving a human-computer dialogue , 1990, CACM.

[3]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[4]  Stephen M Downs,et al.  The HealthPia GlucoPack Diabetes phone: a usability study. , 2007, Diabetes technology & therapeutics.

[5]  AsbjØRn F⊘lstad,et al.  Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems , 2007 .

[6]  Ali Sunyaev,et al.  Usability of Patient-Centered Health IT: Mixed-Methods Usability Study of ePill , 2014, eHealth.

[7]  Gunnar Hartvigsen,et al.  Mobile Phone-Based Self-Management Tools for Type 2 Diabetes: The Few Touch Application , 2010, Journal of diabetes science and technology.

[8]  Suzanne Bakken,et al.  A Comparison of Usability Evaluation Methods: Heuristic Evaluation versus End-User Think-Aloud Protocol - An Example from a Web-based Communication Tool for Nurse Scheduling , 2009, AMIA.

[9]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.

[10]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[11]  Jason C. Goldwater.  Human Factors and Usability in Mobile Health Design – Factors for Sustained Patient Engagement in Diabetes Care , 2014 .

[12]  Jacob O. Wobbrock,et al.  Understanding usability practices in complex domains , 2010, CHI.

[13]  A. Kushniruk,et al.  Towards an Integrative Cognitive-Socio-Technical Approach in Health Informatics: Analyzing Technology-Induced Error Involving Health Information Systems to Improve Patient Safety , 2010, The open medical informatics journal.

[14]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[15]  Norman Fleischer,et al.  The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. The Diabetes Control and Complications Trial Research Group. , 1993 .

[16]  L. Tarassenko,et al.  Implementation of telehealth support for patients with type 2 diabetes using insulin treatment: an exploratory study. , 2009, Informatics in primary care.

[17]  K. Jones,et al.  Using mobile phones and short message service to deliver self-management interventions for chronic conditions: a meta-review. , 2014, Worldviews on evidence-based nursing.

[18]  Elizabeth M Borycki,et al.  Ensuring the safety of health information systems: using heuristics for patient safety. , 2009, Healthcare quarterly.

[19]  Suk-Il Kim,et al.  Effectiveness of mobile and internet intervention in patients with obese type 2 diabetes , 2008, Int. J. Medical Informatics.

[20]  Asbjørn Følstad,et al.  Work-Domain Experts as Evaluators: Usability Inspection of Domain-Specific Work-Support Systems , 2007, Int. J. Hum. Comput. Interact..

[21]  Jacqueline Brodie,et al.  Extending the Heuristic Evaluation Method through Contextualisation , 2002 .

[22]  J. Stoker,et al.  The Department of Health and Human Services. , 1999, Home healthcare nurse.

[23]  Emily S. Patterson,et al.  (NISTIR 7804) Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records , 2012 .

[24]  D. Turner-Bowker,et al.  Heuristic evaluation and usability testing of a computerized patient-reported outcomes survey for headache sufferers. , 2011, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[25]  Barrett S. Caldwell,et al.  An Empirical Study of Usability Testing: Heuristic Evaluation Vs. User Testing , 2002 .

[26]  Jiajie Zhang,et al.  Applying heuristic evaluation to improve the usability of a telemedicine system. , 2006, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[27]  Kevin B. Johnson,et al.  A pilot test of a tailored mobile and web-based diabetes messaging system for adolescents , 2012, Journal of telemedicine and telecare.

[28]  Vimla L. Patel,et al.  Heuristic evaluation of paper-based Web pages: A simplified inspection usability methodology , 2006, J. Biomed. Informatics.

[29]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[30]  Joseph L. Kannry,et al.  Emerging Approaches to Usability Evaluation of Health Information Systems: Towards In-Situ Analysis of Complex Healthcare Systems and Environments , 2011, MIE.

[31]  Vimla L. Patel,et al.  Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units , 2004, Int. J. Medical Informatics.

[32]  S. Genuth,et al.  The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. , 1993, The New England journal of medicine.

[33]  Ioannis N. Kouris,et al.  Mobile phone technologies and advanced data analysis towards the enhancement of diabetes self-management , 2010, Int. J. Electron. Heal..