Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics

Abstract Objective Mobile health (mHealth) systems are becoming more common for chronic disease management, but usability studies are still needed on patients’ perspectives and mHealth interaction performance. This deficiency is addressed by our quantitative usability study of a mHealth diabetes system evaluating patients’ task performance, satisfaction, and the relationship of these measures to user characteristics. Materials and Methods We used metrics in the International Organization for Standardization (ISO) 9241-11 standard. After standardized training, 10 patients performed representative tasks and were assessed on individual task success, errors, efficiency (time on task), satisfaction (System Usability Scale [SUS]) and user characteristics. Results Tasks of exporting and correcting values proved the most difficult, had the most errors, the lowest task success rates, and consumed the longest times on task. The average SUS satisfaction score was 80.5, indicating good but not excellent system usability. Data trends showed males were more successful in task completion, and younger participants had higher performance scores. Educational level did not influence performance, but a more recent diabetes diagnosis did. Patients with more experience in information technology (IT) also had higher performance rates. Discussion Difficult task performance indicated areas for redesign. Our methods can assist others in identifying areas in need of improvement. Data about user background and IT skills also showed how user characteristics influence performance and can provide future considerations for targeted mHealth designs. Conclusion Using the ISO 9241-11 usability standard, the SUS instrument for satisfaction and measuring user characteristics provided objective measures of patients’ experienced usability. These could serve as an exemplar for standardized, quantitative methods for usability studies on mHealth systems.

[1]  Da Tao,et al.  Usability Study of a Computer-Based Self-Management System for Older Adults with Chronic Diseases , 2012, JMIR research protocols.

[2]  Mattias Georgsson,et al.  Going Mobile With Diabetes Support: A Randomized Study of a Text Message–Based Personalized Behavioral Intervention for Type 2 Diabetes Self-Care , 2015, Diabetes Spectrum.

[3]  Ioannis N. Kouris,et al.  Mobile phone technologies and advanced data analysis towards the enhancement of diabetes self-management , 2010, Int. J. Electron. Heal..

[4]  Gerrit C. van der Veer,et al.  Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems , 1993 .

[5]  Seth Wolpin,et al.  Usability evaluation of the interactive Personal Patient Profile-Prostate decision support system with African American men. , 2010, Journal of the National Medical Association.

[6]  P. Timsina,et al.  Mobile Applications for Diabetes Self-Management: Status and Potential , 2013, Journal of diabetes science and technology.

[7]  C. Lyles,et al.  Getting a Technology-Based Diabetes Intervention Ready for Prime Time: a Review of Usability Testing Studies , 2014, Current Diabetes Reports.

[8]  Joseph Sharit,et al.  Factors influencing use of an e-health website in a community sample of older adults , 2013, J. Am. Medical Informatics Assoc..

[9]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[10]  Lucy Yardley,et al.  Understanding reactions to an internet-delivered health-care intervention: accommodating user preferences for information provision , 2010, BMC Medical Informatics Decis. Mak..

[11]  Constantinos K. Coursaris,et al.  A Meta-Analytical Review of Empirical Mobile Usability Studies , 2011 .

[12]  C. Quinn,et al.  Patient Understanding of Diabetes Self-Management: Participatory Decision-Making in Diabetes Care , 2011, Journal of diabetes science and technology.

[13]  Claude Ghaoui,et al.  Encyclopedia of Human Computer Interaction , 2005 .

[14]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[15]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[16]  Robert A. Virzi,et al.  Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough? , 1992 .

[17]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[18]  William G. Griswold,et al.  Usability and Feasibility of PmEB: A Mobile Phone Application for Monitoring Real Time Caloric Balance , 2006, 2006 Pervasive Health Conference and Workshops.

[19]  Kasper Hornbæk,et al.  Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? , 2000, CHI.

[20]  Lee M Ritterband,et al.  Mobile Intervention Design in Diabetes: Review and Recommendations , 2011, Current diabetes reports.

[21]  Dag Svanæs,et al.  The Contextual Nature of Usability and its Relevance to Medical Informatics , 2008, MIE.

[22]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[23]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.

[24]  C. Quinn,et al.  Cluster-Randomized Trial of a Mobile Phone Personalized Behavioral Intervention for Blood Glucose Control , 2011, Diabetes Care.

[25]  Nigel Bevan,et al.  International standards for usability should be more widely used , 2009 .

[26]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[27]  Gunnar Hartvigsen,et al.  Mobile Phone-Based Self-Management Tools for Type 2 Diabetes: The Few Touch Application , 2010, Journal of diabetes science and technology.

[28]  Nigel Bevan,et al.  International Standards for HCI , 2006 .

[29]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[30]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[31]  Joseph L. Kannry,et al.  Emerging Approaches to Usability Evaluation of Health Information Systems: Towards In-Situ Analysis of Complex Healthcare Systems and Environments , 2011, MIE.

[32]  Prem Timsina,et al.  A systematic review of IT for diabetes self-management: Are we there yet? , 2013, Int. J. Medical Informatics.

[33]  CC Quinn,et al.  Cluster-randomized trial of a mobile phone personalized behavioral intervention for blood glucose control. Diabetes Care 2011;34:1934–1942 , 2013, Diabetes Care.

[34]  Erik Liljegren,et al.  Usability in a medical technology context assessment of methods for usability evaluation of medical equipment , 2006 .

[35]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[36]  Martin C. Maguire,et al.  Methods to support human-centred design , 2001, Int. J. Hum. Comput. Stud..

[37]  J. Powell,et al.  Systematic review of communication technologies to promote access and engagement of young people with diabetes into healthcare , 2011, BMC endocrine disorders.

[38]  K. Jones,et al.  Using mobile phones and short message service to deliver self-management interventions for chronic conditions: a meta-review. , 2014, Worldviews on evidence-based nursing.

[39]  Robert L. Wears,et al.  Health information technology: fallacies and sober realities , 2010, J. Am. Medical Informatics Assoc..

[40]  J. B. Brooke,et al.  SUS: a retrospective , 2013 .

[41]  Diane Gromala,et al.  Usability testing of ANSWER: a web-based methotrexate decision aid for patients with rheumatoid arthritis , 2013, BMC Medical Informatics and Decision Making.