Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center

BACKGROUND  Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups. OBJECTIVES  We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard. METHODS  An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data. RESULTS  The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models. CONCLUSION  We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.

[1]  Jacqueline A Merrill,et al.  The Development of Heuristics for Evaluation of Dashboard Visualizations , 2018, Applied Clinical Informatics.

[2]  Mustafa Y. Sir,et al.  Development of data integration and visualization tools for the Department of Radiology to display operational and strategic metrics , 2018, AMIA.

[3]  Barry Aaronson,et al.  Effect of a Real-Time Electronic Dashboard on a Rapid Response System , 2017, Journal of Medical Systems.

[4]  David T. Bauer,et al.  The design and evaluation of a graphical display for laboratory data , 2010, J. Am. Medical Informatics Assoc..

[5]  Michael G Kahn,et al.  Quantifying clinical data quality using relative gold standards. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[6]  Kavita Radhakrishnan,et al.  Visual Analytics for Pattern Discovery in Home Care , 2016, Applied Clinical Informatics.

[7]  T. Lang,et al.  Timely Data for Targeted Quality Improvement Interventions: Use of a Visual Analytics Dashboard for Bronchiolitis , 2019, Applied Clinical Informatics.

[8]  Approaches to Medical Decision-Making Based on Big Clinical Data , 2018, Journal of healthcare engineering.

[9]  Moira Maguire,et al.  Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. , 2017 .

[10]  Hai Hu,et al.  QAIT: A quality assurance issue tracking tool to facilitate the improvement of clinical data quality , 2013, Comput. Methods Programs Biomed..

[11]  Reza Safdari,et al.  From Information Management to Information Visualization , 2016, Applied Clinical Informatics.

[12]  Jo Wood,et al.  On the reliability of usability testing , 2001, CHI Extended Abstracts.

[13]  D. Norman.  The Design of Everyday Things: Revised and Expanded Edition , 2013 .

[14]  Timothy C. Huber,et al.  Developing an Interactive Data Visualization Tool to Assess the Impact of Decision Support on Clinical Operations , 2018, J. Digit. Imaging.

[15]  William A. Knaus,et al.  Implementation Brief: A User-centered Model for Web Site Design: Needs Assessment, User Interface Design, and Rapid Prototyping , 2002, J. Am. Medical Informatics Assoc..

[16]  Nancy Staggers,et al.  Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics , 2015, J. Am. Medical Informatics Assoc..

[17]  Jorge A. Gálvez,et al.  Design and Implementation of a Visual Analytics Electronic Antibiogram within an Electronic Health Record System at a Tertiary Pediatric Hospital , 2018, Applied Clinical Informatics.

[18]  Julia Adler-Milstein,et al.  HITECH Act Drove Large Gains In Hospital Electronic Health Record Adoption. , 2017, Health affairs.

[19]  Gheorghita Ghinea,et al.  Mobile three-dimensional visualisation technologies for clinician-led fall prevention assessments , 2019, Health Informatics J..

[20]  Daniel Weiskopf,et al.  Visual Analytics for Mobile Eye Tracking , 2017, IEEE Transactions on Visualization and Computer Graphics.

[21]  Gang Luo,et al.  MLBCD: a machine learning tool for big clinical data , 2015, Health Information Science and Systems.

[22]  Niels De Ruiter,et al.  A Hybrid 2D/3D User Interface for Radiological Diagnosis , 2018, Journal of Digital Imaging.

[23]  Michael Zeiler,et al.  Usability and Engagement Evaluation of an Unguided Online Program for Promoting a Healthy Lifestyle and Reducing the Risk for Eating Disorders and Obesity in the School Setting , 2019, Nutrients.

[24]  Jan H. Spyridakis,et al.  The relevance of reliability and validity to usability testing , 1989 .

[25]  Harry Hochheiser,et al.  Evaluating visual analytics for health informatics applications: a systematic review from the American Medical Informatics Association Visual Analytics Working Group Task Force on Evaluation , 2019, J. Am. Medical Informatics Assoc..

[26]  Xing He,et al.  ALOHA: developing an interactive graph-based visualization for dietary supplement knowledge graph through user-centered design , 2019, BMC Medical Informatics and Decision Making.

[27]  Austin M. Coe,et al.  Usability Testing of a Web-Based Decision Aid for Breast Cancer Risk Assessment Among Multi-Ethnic Women , 2016, AMIA.

[28]  Jesus J. Caban,et al.  Visual analytics in healthcare - opportunities and research challenges , 2015, J. Am. Medical Informatics Assoc..

[29]  Hans-Ulrich Prokosch,et al.  Evaluation of an Interactive Visualization Tool for the Interpretation of Pediatric Laboratory Test Results , 2017, GMDS.

[30]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[31]  R. D. de Bie,et al.  Biofeedback in Partial Weight Bearing: Usability of Two Different Devices from a Patient’s and Physical Therapist’s Perspective , 2016, PloS one.

[32]  Bernd Ploderer,et al.  Promoting Self-Care of Diabetic Foot Ulcers Through a Mobile Phone App: User-Centered Design and Evaluation , 2018, JMIR diabetes.

[33]  David Borland,et al.  Data-Driven Healthcare: Challenges and Opportunities for Interactive Visualization , 2016, IEEE Computer Graphics and Applications.

[34]  Ben Shneiderman,et al.  Novel user interface design for medication reconciliation: an evaluation of Twinlist , 2015, J. Am. Medical Informatics Assoc..