Heuristic evaluation and user tests of wearable mobile health monitoring applications: what results do different methods yield?

Wearable smart bands connected to mobile apps in smartphones provide essential resources for health and activity monitoring. Evaluating this kind of application has many challenges, considering the mobile and wearable nature. This paper presents a comparative study employing heuristic evaluations and user tests of three health monitoring apps used with their wearable smart bands, to analyse the types of problems identified by each method. The study examined the Mi Band 2 with the MiFit app, the MTR-01 with the DayDay Band app, and Goral Y5 with the Lefun app. The apps and wearable smart bands were evaluated by three inspectors and employing user tests in a laboratory with a panel of 11 users simulating a short walk. After mapping similar problems encountered, the study found 18 problems by both methods, 27 problems encountered only by users, and 105 problems encountered only by the heuristic evaluations. The results in this study are relevant to inform designers about the types of problems in wearable health apps and devices using different methods, and research gaps to improve usability methods and resources to strengthen evaluations in this domain.

[1]  Antonella De Angeli,et al.  Co-designing wearable devices for sports: The case study of sport climbing , 2019, Int. J. Hum. Comput. Stud..

[2]  Hao Luo,et al.  The impact of industrial wearable system on industry 4.0 , 2018, 2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC).

[3]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[4]  Vivian Genaro Motti,et al.  Human Factors Considerations in the Design of Wearable Devices , 2014 .

[5]  P. Bonato,et al.  From A to Z: Wearable technology explained. , 2018, Maturitas.

[6]  Mohammad Hossein Jarrahi,et al.  Wearable activity trackers, accuracy, adoption, acceptance and health impact: A systematic literature review , 2019, J. Biomed. Informatics.

[7]  Michael A. Rupp,et al.  The role of individual differences on perceptions of wearable fitness device trust, usability, and motivational impact. , 2018, Applied ergonomics.

[8]  Karolina Baras,et al.  Assessment of activity trackers: toward an acceptance model , 2016, UbiComp Adjunct.

[9]  Patrick Wahl,et al.  Criterion-Validity of Commercially Available Physical Activity Tracker to Estimate Step Count, Covered Distance and Energy Expenditure during Sports Conditions , 2017, Front. Physiol..

[10]  Jakob Nielsen,et al.  Improving a human-computer dialogue , 1990, CACM.

[11]  Rajiv Sabherwal,et al.  Mobile technology identity and self-efficacy: Implications for the adoption of clinically supported mobile health apps , 2019, Int. J. Inf. Manag..

[12]  Helen Petrie,et al.  What do users really care about?: a comparison of usability problems found by users and experts on highly interactive websites , 2012, CHI.

[13]  Vivian Genaro Motti,et al.  An overview of wearable applications for healthcare: requirements and challenges , 2015, UbiComp/ISWC Adjunct.

[14]  Matthew S. Eastin,et al.  Wearable fitness technology: A structural investigation into acceptance and perceived fitness outcomes , 2016, Comput. Hum. Behav..

[15]  Glyn Lawson,et al.  Establishing usability heuristics for heuristics evaluation in a specific domain: Is there a consensus? , 2016, Applied ergonomics.

[16]  Junqing Xie,et al.  Evaluating the Validity of Current Mainstream Wearable Devices in Fitness Tracking Under Various Physical Activities: Comparative Study , 2018, JMIR mHealth and uHealth.

[17]  Samuel Ssemugabi,et al.  A comparative study of two usability evaluation methods using a web-based e-learning application , 2007, SAICSIT '07.

[18]  Mary F Lesch,et al.  Employee acceptance of wearable technology in the workplace. , 2019, Applied ergonomics.

[19]  M. McNarry,et al.  Feasibility and Effectiveness of Using Wearable Activity Trackers in Youth: A Systematic Review , 2016, JMIR mHealth and uHealth.

[20]  Henry Been-Lirn Duh,et al.  Usability evaluation for mobile device: a comparison of laboratory and field tests , 2006, Mobile HCI.