Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process

Background  With the increased usage of dashboard reporting systems to monitor and track patient panels by clinical users, developers must ensure that the information displays they produce are accurate and intuitive. When evaluating usability of a clinical dashboard among potential end users, developers oftentimes rely on methods such as questionnaires as opposed to other, more time-intensive strategies that incorporate direct observation. Objectives  Prior to release of the potentially inappropriate medication (PIM) clinical dashboard, designed to facilitate completion of a quality improvement project by clinician scholars enrolled in the Veterans Affairs (VA) workforce development Geriatric Scholars Program (GSP), we evaluated the usability of the system. This article describes the process of usability testing a dashboard reporting system with clinicians using direct observation and think-aloud moderating techniques. Methods  We developed a structured interview protocol that combines virtual observation, think-aloud moderating techniques, and retrospective questioning of the overall user experience, including use of the System Usability Scale (SUS). Thematic analysis was used to analyze field notes from the interviews of three GSP alumni. Results  Our structured approach to usability testing identified specific functional problems with the dashboard reporting system that were missed by results from the SUS. Usability testing lead to overall improvements in the intuitive use of the system, increased data transparency, and clarification of the dashboard's purpose. Conclusion  Reliance solely on questionnaires and surveys at the end stages of dashboard development can mask potential functional problems that will impede proper usage and lead to misinterpretation of results. A structured approach to usability testing in the developmental phase is an important tool for developers of clinician friendly systems for displaying easily digested information and tracking outcomes for the purpose of quality improvement.

[1]  Patricia C Dykes,et al.  User-Centered Collaborative Design and Development of an Inpatient Safety Dashboard. , 2017, Joint Commission journal on quality and patient safety.

[2]  Sooyoung Yoo,et al.  A novel concept for integrating and delivering health information using a comprehensive digital dashboard: An analysis of healthcare professionals' intention to adopt a new system and the trend of its real usage , 2017, Int. J. Medical Informatics.

[3]  Noah M Ivers,et al.  Using report cards and dashboards to drive quality improvement: lessons learnt and lessons still to learn , 2018, BMJ Quality & Safety.

[4]  J. Wyatt,et al.  Evaluating computerised health information systems: hard lessons still to be learnt , 2003, BMJ : British Medical Journal.

[5]  Jianwei Leng,et al.  VA Geriatric Scholars Program's impact on prescribing potentially inappropriate medications. , 2019, The American journal of managed care.

[6]  Marie T. Egan,et al.  Clinical Dashboards: Impact on Workflow, Care Quality, and Patient Safety , 2006, Critical care nursing quarterly.

[7]  Amanda Nolen,et al.  Thematic analysis of qualitative research data: Is it as easy as it sounds? , 2018, Currents in pharmacy teaching & learning.

[8]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[9]  Raj M. Ratwani,et al.  A Decade of Health Information Technology Usability Challenges and the Path Forward , 2019, JAMA.

[10]  S. Barczi,et al.  Veterans Affairs Geriatric Scholars Program: Enhancing Existing Primary Care Clinician Skills in Caring for Older Veterans , 2016, Journal of the American Geriatrics Society.

[11]  Katharine T. Adams,et al.  Electronic Health Record Usability Issues and Potential Contribution to Patient Harm , 2018, JAMA.

[12]  Walter V. Sujansky,et al.  Heterogeneous Database Integration in Biomedicine , 2001, J. Biomed. Informatics.

[13]  Reza Safdari,et al.  Development of Performance Dashboards in Healthcare Sector: Key Practical Issues , 2015, Acta informatica medica : AIM : journal of the Society for Medical Informatics of Bosnia & Herzegovina : casopis Drustva za medicinsku informatiku BiH.

[14]  K. A. Ericsson,et al.  Protocol analysis: Verbal reports as data, Rev. ed. , 1993 .

[15]  R. Haynes,et al.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. , 1998, JAMA.

[16]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[17]  Ashley N. D. Meyer,et al.  The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations , 2014, BMJ quality & safety.

[18]  Inocencio Maramba,et al.  Methods of usability testing in the development of eHealth applications: A scoping review , 2019, Int. J. Medical Informatics.

[19]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[20]  Jacqueline A Merrill,et al.  Usability Evaluation of a Dashboard for Home Care Nurses , 2019, Computers, informatics, nursing : CIN.

[21]  J. M. Christian Bastien,et al.  Usability testing: a review of some methodological and technical aspects of the method , 2010, Int. J. Medical Informatics.

[22]  A. Darzi,et al.  Systematic review of the application of the plan–do–study–act method to improve quality in healthcare , 2013, BMJ quality & safety.

[23]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[24]  Ganesh Bhutkar,et al.  A review: healthcare usability evaluation methods. , 2013, Biomedical instrumentation & technology.

[25]  Jacqueline A Merrill,et al.  The Development of Heuristics for Evaluation of Dashboard Visualizations , 2018, Applied Clinical Informatics.

[26]  Sandra Dunn,et al.  A mixed methods evaluation of the maternal-newborn dashboard in Ontario: dashboard attributes, contextual factors, and facilitators and barriers to use: a study protocol , 2015, Implementation Science.

[27]  Ali Jalali,et al.  Quality Initiative Using Theory of Change and Visual Analytics to Improve Controlled Substance Documentation Discrepancies in the Operating Room , 2019, Applied Clinical Informatics.