Usability Study of Mainstream Wearable Fitness Devices: Feature Analysis and System Usability Scale Evaluation

Background Wearable devices have the potential to promote a healthy lifestyle because of their real-time data monitoring capabilities. However, device usability is a critical factor that determines whether they will be adopted on a large scale. Usability studies on wearable devices are still scarce. Objective This study aims to compare the functions and attributes of seven mainstream wearable devices and to evaluate their usability. Methods The wearable devices selected were the Apple Watch, Samsung Gear S, Fitbit Surge, Jawbone Up3, Mi Band, Huawei Honor B2, and Misfit Shine. A mixed method of feature comparison and a System Usability Scale (SUS) evaluation based on 388 participants was applied; the higher the SUS score, the better the usability of the product. Results For features, all devices had step counting, an activity timer, and distance recording functions. The Samsung Gear S had a unique sports track recording feature and the Huawei Honor B2 had a unique wireless earphone. The Apple Watch, Samsung Gear S, Jawbone Up3, and Fitbit Surge could measure heart rate. All the devices were able to monitor sleep, except the Apple Watch. For product characteristics, including attributes such as weight, battery life, price, and 22 functions such as step counting, activity time, activity type identification, sleep monitoring, and expandable new features, we found a very weak negative correlation between the SUS scores and price (r=−.10, P=.03) and devices that support expandable new features (r=−.11, P=.02), and a very weak positive correlation between the SUS scores and devices that support the activity type identification function (r=.11, P=.02). The Huawei Honor B2 received the highest score of mean 67.6 (SD 16.1); the lowest Apple Watch score was only 61.4 (SD 14.7). No significant difference was observed among brands. The SUS score had a moderate positive correlation with the user’s experience (length of time the device was used) (r=.32, P<.001); participants in the medical and health care industries gave a significantly higher score (mean 61.1, SD 17.9 vs mean 68.7, SD 14.5, P=.03). Conclusions The functions of wearable devices tend to be homogeneous and usability is similar across various brands. Overall, Mi Band had the lowest price and the lightest weight. Misfit Shine had the longest battery life and most functions, and participants in the medical and health care industries had the best evaluation of wearable devices. The perceived usability of mainstream wearable devices is unsatisfactory and customer loyalty is not high. A consumer’s SUS rating for a wearable device is related to their personal situation instead of the device brand. Device manufacturers should put more effort into developing innovative functions and improving the usability of their products by integrating more cognitive behavior change techniques.

[1]  T. Oh,et al.  Clinical Feasibility of Continuously Monitored Data for Heart Rate, Physical Activity, and Sleeping by Wearable Activity Trackers in Patients with Thyrotoxicosis: Protocol for a Prospective Longitudinal Observational Study , 2018, JMIR research protocols.

[2]  Lieveke Ameye,et al.  Reliability of commercially available sleep and activity trackers with manual switch-to-sleep mode activation in free-living healthy individuals , 2017, Int. J. Medical Informatics.

[3]  Hyunwoo Lee,et al.  An Enhanced Method to Estimate Heart Rate from Seismocardiography via Ensemble Averaging of Body Movements at Six Degrees of Freedom , 2018, Sensors.

[4]  Winnie Bell,et al.  Scaling up Dietary Data for Decision-Making in Low-Income Countries: New Technological Frontiers , 2017, Advances in nutrition.

[5]  Zhaohui Wu,et al.  Infrastructure and Reliability Analysis of Electric Networks for E-Textiles , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[6]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[7]  R. Davey,et al.  Physical activity in relation to urban environments in 14 cities worldwide: a cross-sectional study , 2016, The Lancet.

[8]  Arash Naeim,et al.  Telehealth in older adults with cancer in the United States: The emerging use of wearable sensors. , 2017, Journal of geriatric oncology.

[9]  C. Abraham,et al.  The Behavior Change Technique Taxonomy (v1) of 93 Hierarchically Clustered Techniques: Building an International Consensus for the Reporting of Behavior Change Interventions , 2013, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[10]  Kanitthika Kaewkannate,et al.  A comparison of wearable fitness devices , 2016, BMC Public Health.

[11]  Dong Wen,et al.  Evaluating the Consistency of Current Mainstream Wearable Devices in Health Monitoring: A Comparison Under Free-Living Conditions , 2017, Journal of medical Internet research.

[12]  Himalaya Patel,et al.  An Improved Usability Measure Based on Novice and Expert Performance , 2011, Int. J. Hum. Comput. Interact..

[13]  James M. Walker,et al.  Measurement of CPOE end-user satisfaction among ICU physicians and nurses , 2010, Applied Clinical Informatics.

[14]  R. Kato,et al.  Clinical Usefulness of Wearable Cardioverter Defibrillator (WCD) and Current Understanding of Its Clinical Indication in Japan. , 2018, Circulation journal : official journal of the Japanese Circulation Society.

[15]  Junqing Xie,et al.  Evaluating the Validity of Current Mainstream Wearable Devices in Fitness Tracking Under Various Physical Activities: Comparative Study , 2018, JMIR mHealth and uHealth.

[16]  Sundaresan Jayaraman,et al.  A transdisciplinary approach to wearables, big data and quality of life , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  B. Fjeldsoe,et al.  Usage, Acceptability, and Effectiveness of an Activity Tracker in a Randomized Trial of a Workplace Sitting Intervention: Mixed-Methods Evaluation , 2018, Interactive journal of medical research.

[18]  Jeff Sauro,et al.  The Factor Structure of the System Usability Scale , 2009, HCI.

[19]  R. Fielding,et al.  A Study of the Reliability and Concurrent Validity of the Chinese Version of the Pain Medication Attitude Questionnaire (ChPMAQ) in a Sample of Chinese Patients with Chronic Pain. , 2016, Pain medicine.

[20]  Mary K Goldstein,et al.  Evaluation of the acceptability and usability of a decision support system to encourage safe and effective use of opioid therapy for chronic, noncancer pain by primary care providers. , 2010, Pain medicine.

[21]  Ronald E. Gangnon,et al.  The Accuracy of Heart Rate Monitoring by Some Wrist-Worn Activity Trackers. , 2017, Annals of internal medicine.

[22]  Elizabeth M. Borycki,et al.  Physician satisfaction with a critical care clinical information system using a multimethod evaluation of usability , 2018, Int. J. Medical Informatics.

[23]  J. Schneider,et al.  Cognitive Behavioral Therapy to Promote Exercise Behavior in Older Adults: Implications for Physical Therapists , 2005, Journal of geriatric physical therapy.

[24]  Aydogan Ozcan,et al.  Wearable and Implantable Sensors for Biomedical Applications. , 2018, Annual review of analytical chemistry.

[25]  P. Ray,et al.  Health Care Provider Adoption of eHealth: Systematic Literature Review , 2013, Interactive journal of medical research.

[26]  Kate Lorig,et al.  Internet Versus Mailed Questionnaires: A Randomized Comparison , 2004, Journal of medical Internet research.

[27]  Michael D. Byrne,et al.  A Comparison of Usability Between Voting Methods , 2006, EVT.

[28]  Dong Wen,et al.  Consumers' perceived attitudes to wearable devices in health monitoring in China: A survey study , 2017, Comput. Methods Programs Biomed..

[29]  Reza Khajouei,et al.  Comparison of two heuristic evaluation methods for evaluating the usability of health information systems , 2018, J. Biomed. Informatics.

[30]  Min Zhang,et al.  Flexible, Stretchable Sensors for Wearable Health Monitoring: Sensing Mechanisms, Materials, Fabrication Strategies and Features , 2018, Sensors.

[31]  Matthew P Buman,et al.  Wearable Technology and Physical Activity in Chronic Disease: Opportunities and Challenges. , 2018, American journal of preventive medicine.

[32]  D. Asch,et al.  Wearable devices as facilitators, not drivers, of health behavior change. , 2015, JAMA.

[33]  Joseph Peter Salisbury,et al.  Concussion Assessment With Smartglasses: Validation Study of Balance Measurement Toward a Lightweight, Multimodal, Field-Ready Platform , 2018, JMIR mHealth and uHealth.

[34]  Stephan Selinger,et al.  A Comparison of Multiple Wearable Devices Regarding their User Experience During Running , 2016 .

[35]  Grace W K Ho,et al.  Examining Perceptions and Attitudes , 2017, Western journal of nursing research.

[36]  M. Hassenzahl,et al.  Wellbeing in the Making: Peoples’ Experiences with Wearable Activity Trackers , 2016, Psychology of well-being.

[37]  Jurek Kirakowski,et al.  Measuring the Usability of Web Sites , 1998 .

[38]  Reza Khajouei,et al.  Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems , 2017, J. Am. Medical Informatics Assoc..

[39]  Peter S Lum,et al.  Pilot testing of the spring operated wearable enhancer for arm rehabilitation (SpringWear) , 2018, Journal of NeuroEngineering and Rehabilitation.

[40]  William Lidwell,et al.  Universal Principles of Design , 2003 .

[41]  Daniel R. Bateman,et al.  Categorizing Health Outcomes and Efficacy of mHealth Apps for Persons With Cognitive Impairment: A Systematic Review , 2017, Journal of medical Internet research.

[42]  Susan Harker,et al.  ISO 9241-11 Revised: What Have We Learnt About Usability Since 1998? , 2015, HCI.

[43]  Philip T. Kortum,et al.  Usability Ratings for Everyday Products Measured With the System Usability Scale , 2013, Int. J. Hum. Comput. Interact..

[44]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[45]  S. Alfonsson,et al.  Interformat Reliability of Digital Psychiatric Self-Report Questionnaires: A Systematic Review , 2014, Journal of medical Internet research.

[46]  M. Lachman,et al.  Behavior Change with Fitness Technology in Sedentary Adults: A Review of the Evidence for Increasing Physical Activity , 2017, Front. Public Health.

[47]  Taeghwan Hyeon,et al.  Enzyme‐Based Glucose Sensor: From Invasive to Wearable Device , 2018, Advanced healthcare materials.

[48]  Cindy M. Gray,et al.  Evaluating the Impact of Physical Activity Apps and Wearables: Interdisciplinary Review , 2018, JMIR mHealth and uHealth.

[49]  Michael D. Byrne,et al.  Voting on a Smartphone , 2011 .

[50]  Liza S. Rovniak,et al.  Engineering Online and In-Person Social Networks for Physical Activity: A Randomized Trial , 2016, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[51]  R. Gurrea,et al.  The role played by perceived usability, satisfaction and consumer trust on website loyalty , 2006, Inf. Manag..

[52]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[53]  Parmit K. Chilana,et al.  Acceptance of Commercially Available Wearable Activity Trackers Among Adults Aged Over 50 and With Chronic Illness: A Mixed-Methods Evaluation , 2016, JMIR mHealth and uHealth.