Analysis of a Web-Based Dashboard to Support the Use of National Audit Data in Quality Improvement: Realist Evaluation

Background Dashboards can support data-driven quality improvements in health care. They visualize data in ways intended to ease cognitive load and support data comprehension, but how they are best integrated into working practices needs further investigation. Objective This paper reports the findings of a realist evaluation of a web-based quality dashboard (QualDash) developed to support the use of national audit data in quality improvement. Methods QualDash was co-designed with data users and installed in 8 clinical services (3 pediatric intensive care units and 5 cardiology services) across 5 health care organizations (sites A-E) in England between July and December 2019. Champions were identified to support adoption. Data to evaluate QualDash were collected between July 2019 and August 2021 and consisted of 148.5 hours of observations including hospital wards and clinical governance meetings, log files that captured the extent of use of QualDash over 12 months, and a questionnaire designed to assess the dashboard’s perceived usefulness and ease of use. Guided by the principles of realist evaluation, data were analyzed to understand how, why, and in what circumstances QualDash supported the use of national audit data in quality improvement. Results The observations revealed that variation across sites in the amount and type of resources available to support data use, alongside staff interactions with QualDash, shaped its use and impact. Sites resourced with skilled audit support staff and established reporting systems (sites A and C) continued to use existing processes to report data. A number of constraints influenced use of QualDash in these sites including that some dashboard metrics were not configured in line with user expectations and staff were not fully aware how QualDash could be used to facilitate their work. In less well-resourced services, QualDash automated parts of their reporting process, streamlining the work of audit support staff (site B), and, in some cases, highlighted issues with data completeness that the service worked to address (site E). Questionnaire responses received from 23 participants indicated that QualDash was perceived as useful and easy to use despite its variable use in practice. Conclusions Web-based dashboards have the potential to support data-driven improvement, providing access to visualizations that can help users address key questions about care quality. Findings from this study point to ways in which dashboard design might be improved to optimize use and impact in different contexts; this includes using data meaningful to stakeholders in the co-design process and actively engaging staff knowledgeable about current data use and routines in the scrutiny of the dashboard metrics and functions. In addition, consideration should be given to the processes of data collection and upload that underpin the quality of the data visualized and consequently its potential to stimulate quality improvement. International Registered Report Identifier (IRRID) RR2-10.1136/bmjopen-2019-033208

[1]  D. Ivanković,et al.  Features Constituting Actionable COVID-19 Dashboards: Descriptive Assessment and Expert Appraisal of 158 Public Web-Based COVID-19 Dashboards , 2020, Journal of medical Internet research.

[2]  Junzhang Tian,et al.  Digital Health Technologies Respond to the COVID-19 Pandemic In a Tertiary Hospital in China: Development and Usability Study , 2020, Journal of medical Internet research.

[3]  P. Doherty,et al.  Exploring variation in the use of feedback from national clinical audits: a realist investigation , 2020, BMC Health Services Research.

[4]  Roy A. Ruddle,et al.  QualDash: Adaptable Generation of Visualisation Dashboards for Healthcare Quality Improvement , 2020, IEEE Transactions on Visualization and Computer Graphics.

[5]  Brian C Sauer,et al.  Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process , 2020, Applied Clinical Informatics.

[6]  M. Cucciniello,et al.  Harnessing Digital Health Technologies During and After the COVID-19 Pandemic: Context Matters , 2020, Journal of medical Internet research.

[7]  Judith W. Dexheimer,et al.  An interactive online dashboard for tracking COVID-19 in U.S. counties, cities, and states in real time , 2020, J. Am. Medical Informatics Assoc..

[8]  P. Doherty,et al.  Institutional use of National Clinical Audits by healthcare providers. , 2020, Journal of evaluation in clinical practice.

[9]  J. Kay,et al.  Developing an Intranet-Based Lymphedema Dashboard for Breast Cancer Multidisciplinary Teams: Design Research Study , 2020, Journal of medical Internet research.

[10]  Sarah Alderson,et al.  Revitalising audit and feedback to improve patient care , 2020, BMJ.

[11]  R. Ruddle,et al.  How, in what contexts, and why do quality dashboards lead to improvements in care quality in acute hospitals? Protocol for a realist feasibility evaluation , 2020, BMJ Open.

[12]  C. Gale,et al.  The Myocardial Ischaemia National Audit Project (MINAP) , 2019 .

[13]  P. Carayon Human Factors in Health(care) Informatics: Toward Continuous Sociotechnical System Design , 2019, CSHI.

[14]  David Russell,et al.  Using Feedback Intervention Theory to Guide Clinical Dashboard Design , 2018, AMIA.

[15]  Saif S. Khairat,et al.  The Impact of Visualization Dashboards on Quality of Care and Clinician Satisfaction: Integrative Literature Review , 2018, JMIR human factors.

[16]  James R. Lewis,et al.  The System Usability Scale: Past, Present, and Future , 2018, Int. J. Hum. Comput. Interact..

[17]  V. Eatough,et al.  Framework analysis: a worked example of a study exploring young people’s experiences of depression , 2016 .

[18]  Jenny Neuburger,et al.  How is feedback from national clinical audits used? Views from English National Health Service trust audit leads , 2016, Journal of health services research & policy.

[19]  Rebecca Randell,et al.  Using realist reviews to understand how health IT works, for whom, and in what circumstances , 2015, J. Am. Medical Informatics Assoc..

[20]  Geraldine Fitzpatrick,et al.  Dashboards for improving patient care: Review of the literature , 2015, Int. J. Medical Informatics.

[21]  Alan Hedge,et al.  Cognitive ergonomics, socio-technical systems, and the impact of healthcare information technologies , 2011 .

[22]  Dean F Sittig,et al.  A new sociotechnical model for studying health information technology in complex adaptive healthcare systems , 2010, Quality and Safety in Health Care.

[23]  Joseph W. Alba,et al.  Heuristics and Biases in Data-Based Decision Making: Effects of Experience, Training, and Graphical Data Displays , 2010 .

[24]  L. Smeeth,et al.  The Myocardial Ischaemia National Audit Project (MINAP) , 2010, Heart.

[25]  B. Karsh,et al.  The Technology Acceptance Model: Its past and its future in health care , 2010, J. Biomed. Informatics.

[26]  A. Srivastava,et al.  Framework Analysis: A Qualitative Methodology for Applied Policy Research , 2009 .

[27]  Alice Percival Healthcare Quality Improvement Partnership , 2008 .

[28]  H. Wunsch,et al.  Variation in critical care services across North America and Western Europe* , 2008, Critical care medicine.

[29]  C. Schoen,et al.  On the front lines of care: primary care doctors' office systems, experiences, and views in seven countries. , 2006, Health affairs.

[30]  L. Spencer,et al.  Qualitative data analysis for applied policy research , 2002 .

[31]  W. M. Fox Sociotechnical System Principles and Guidelines: Past and Present , 1995 .

[32]  Roy A. Ruddle,et al.  Requirements specification for a quality dashboard for exploring National Clinical Audit data , 2019 .

[33]  R. Logie,et al.  When a graph is poorer than 100 words: A comparison of computerised natural language generation, human generated descriptions and graphical displays in neonatal intensive care , 2010 .

[34]  Nick Tilley,et al.  An introduction to scientific realist evaluation , 1997 .

[35]  L. Goldsmith The Qualitative Report The Qualitative Report Using Framework Analysis in Applied Qualitative Research Using Framework Analysis in Applied Qualitative Research , 2022 .