Interface design recommendations for computerised clinical audit and feedback: Hybrid usability evidence from a research-led system

Highlights • Presents a novel theoretically-informed computerised audit and feedback (e-A&F) system.• Identifies usability issues by employing an innovative hybrid inspection approach.• Translates findings into recommendations for the user-centred design of e-A&F systems.

[1]  Yu-Sheng Lo,et al.  Improving the work efficiency of healthcare-associated infection surveillance using electronic medical records , 2014, Comput. Methods Programs Biomed..

[2]  Susan Michie,et al.  Reporting and design elements of audit and feedback interventions: a secondary review , 2016, BMJ Quality & Safety.

[3]  A. Sheikh,et al.  A pharmacist-led information technology intervention for medication errors (PINCER): a multicentre, cluster randomised, controlled trial and cost-effectiveness analysis , 2012, The Lancet.

[4]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[5]  Tom Fahey,et al.  Interventions used to improve control of blood pressure in patients with hypertension. , 2010, The Cochrane database of systematic reviews.

[6]  Jakob Nielsen,et al.  Heuristics for User Interface Design , 2006 .

[7]  Terence S. Andre,et al.  Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems , 2003, Hum. Factors.

[8]  Allan Fong,et al.  'Connecting the dots': leveraging visual analytics to make sense of patient safety event reports , 2015, J. Am. Medical Informatics Assoc..

[9]  Alissa L. Russ,et al.  Recommendations to improve the usability of drug-drug interaction clinical decision support alerts , 2015, J. Am. Medical Informatics Assoc..

[10]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[11]  P. Fayers,et al.  The Visual Display of Quantitative Information , 1990 .

[12]  Charlene R. Weir,et al.  Revisiting Heuristic Evaluation Methods to Improve the Reliability of Findings , 2014, MIE.

[13]  John M. Starmer,et al.  Implementation of a real-time compliance dashboard to help reduce SICU ventilator-associated pneumonia with the ventilator bundle. , 2009, Archives of surgery.

[14]  ShneidermanBen,et al.  Interactive Information Visualization to Explore and Query Electronic Health Records , 2013 .

[15]  Andrew Sears,et al.  Heuristic Walkthroughs: Finding the Problems Without the Noise , 1997, Int. J. Hum. Comput. Interact..

[16]  Morten Hertzum,et al.  The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments , 1998 .

[17]  Rick Spencer,et al.  The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company , 2000, CHI.

[18]  Vijay K. Vaishnavi,et al.  Design Science Research Methods and Patterns: Innovating Information and Communication Technology , 2007 .

[19]  G. Elwyn,et al.  Presenting risk information--a review of the effects of "framing" and other manipulations on patient outcomes. , 2001, Journal of health communication.

[20]  I. James,et al.  Clinical dashboard: use in older adult mental health wards , 2013 .

[21]  Allison B McCoy,et al.  Adopting real-time surveillance dashboards as a component of an enterprisewide medication safety strategy. , 2011, Joint Commission journal on quality and patient safety.

[22]  David W. Bates,et al.  Synthesis of Research Paper: Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-based Medicine a Reality , 2003, J. Am. Medical Informatics Assoc..

[23]  Timothy F. Chen,et al.  An expert panel assessment of comprehensive medication reviews for clients of community mental health teams , 2010, Social Psychiatry and Psychiatric Epidemiology.

[24]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[25]  Nicolette de Keizer,et al.  A Web-based System to Facilitate Local, Systematic Quality Improvement by Multidisciplinary Care Teams: Development and First Experiences of CARDSS Online , 2013, MedInfo.

[26]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[27]  Shobha Phansalkar,et al.  Design of decision support interventions for medication prescribing , 2013, Int. J. Medical Informatics.

[28]  R. Hernaez,et al.  Reliability and agreement studies: a guide for clinical investigators , 2015, Gut.

[29]  Mary Shaw,et al.  Principles for Best Practice in Clinical Audit , 2002 .

[30]  R Brian Haynes,et al.  Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials , 2013, BMJ : British Medical Journal.

[31]  Gustavo Rossi,et al.  An Object Oriented Approach to Web-Based Applications Design , 1998, Theory Pract. Object Syst..

[32]  Karen Tu,et al.  "My approach to this job is...one person at a time": Perceived discordance between population-level quality targets and patient-centred care. , 2014, Canadian family physician Medecin de famille canadien.

[33]  Jeffrey R Curtis,et al.  Challenges in improving the quality of osteoporosis care for long-term glucocorticoid users: a prospective randomized trial. , 2007, Archives of internal medicine.

[34]  Aziz Sheikh,et al.  An embedded longitudinal multi-faceted qualitative evaluation of a complex cluster randomized controlled trial aiming to reduce clinically important errors in medicines management in general practice , 2012, Trials.

[35]  Ben Shneiderman,et al.  Interactive Information Visualization to Explore and Query Electronic Health Records , 2013, Found. Trends Hum. Comput. Interact..

[36]  T Lauritzen,et al.  Improved quality of Type 2 diabetes care following electronic feedback of treatment status to general practitioners: a cluster randomized controlled trial , 2010, Diabetic medicine : a journal of the British Diabetic Association.

[37]  Paul J. Chang,et al.  The Radiology Digital Dashboard: Effects on Report Turnaround Time , 2008, Journal of Digital Imaging.

[38]  R. A. Balk.  Implementation of a Real-time Compliance Dashboard to Help Reduce SICU Ventilator-Associated Pneumonia With the Ventilator Bundle , 2010 .

[39]  William W. Boonn,et al.  Developing an Automated Database for Monitoring Ultrasound- and Computed Tomography-Guided Procedure Complications and Diagnostic Yield , 2013, Journal of Digital Imaging.

[40]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[41]  Erik Duval,et al.  Design and Evaluation of an Interactive Proof-of-Concept Dashboard for General Practitioners , 2015, 2015 International Conference on Healthcare Informatics.

[42]  Iain E. Buchan,et al.  The Case for Conceptual and Computable Cross-Fertilization Between Audit and Feedback and Clinical Decision Support , 2015, MedInfo.

[43]  Diana L Miglioretti,et al.  Web-based mammography audit feedback. , 2012, AJR. American journal of roentgenology.

[44]  Robert J. Kamper.  Extending the Usability of Heuristics for Design and Evaluation: Lead, Follow, and Get Out of the Way , 2002, Int. J. Hum. Comput. Interact..

[45]  Jeffery L. Belden,et al.  A Diabetes Dashboard and Physician Efficiency and Accuracy in Accessing Data Needed for High-Quality Diabetes Care , 2011, The Annals of Family Medicine.

[46]  David R. Flum,et al.  Integrating Patient-Reported Outcomes into Spine Surgical Care through Visual Dashboards: Lessons Learned from Human-Centered Design , 2015, EGEMS.

[47]  Blackford Middleton,et al.  Electronic health record feedback to improve antibiotic prescribing for acute respiratory infections. , 2010, The American journal of managed care.

[48]  Elizabeth M. Borycki,et al.  Integrating Heuristic Evaluation with Cognitive Walkthrough: Development of a Hybrid Usability Inspection Method , 2015, ITCH.

[49]  Douglas S. Bell,et al.  Interface design principles for usable decision support: A targeted review of best practices for clinical prescribing interventions , 2012, J. Biomed. Informatics.

[50]  Bruce Guthrie,et al.  Developing a complex intervention to improve prescribing safety in primary care: mixed methods feasibility and optimisation pilot study , 2014, BMJ Open.

[51]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[52]  Alicia O. Scott-Wright,et al.  A Methodology for Modular Representation of Guidelines , 2004, MedInfo.

[53]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[54]  Timothy F Chen,et al.  Interrater agreement and interrater reliability: key concepts, approaches, and applications. , 2013, Research in social & administrative pharmacy : RSAP.

[55]  L. Hayden,et al.  Ten Commandments for Effective Clinical Decision Support: Making the Practice of Evidence-based Medicine a Reality , 2011 .

[56]  Jane M. Young,et al.  Audit and feedback: effects on professional practice and healthcare outcomes. , 2012, The Cochrane database of systematic reviews.

[57]  E Ammenwerth,et al.  Usability Flaws in Medication Alerting Systems: Impact on Usage and Work System , 2015, Yearbook of Medical Informatics.

[58]  S. Shortell,et al.  Improving the quality of health care in the United Kingdom and the United States: a framework for change. , 2001, The Milbank quarterly.

[59]  Elizabeth M Borycki,et al.  Ensuring the safety of health information systems: using heuristics for patient safety. , 2009, Healthcare quarterly.

[60]  Anthony C. Robinson,et al.  Designing a web-based learning portal for geographic visualization and analysis in public health , 2011, Health Informatics J..

[61]  Jorge A. Gálvez,et al.  Optimization of drug-drug interaction alert rules in a pediatric hospital's electronic health record system using a visual analytics dashboard , 2015, J. Am. Medical Informatics Assoc..

[62]  Claudia Pagliari,et al.  DARTS 2000 online diabetes management system: formative evaluation in clinical practice. , 2003, Journal of evaluation in clinical practice.

[63]  A Kushniruk,et al.  Usability Methods for Ensuring Health Information Technology Safety: Evidence-Based Approaches Contribution of the IMIA Working Group Health Informatics for Patient Safety , 2013, Yearbook of Medical Informatics.

[64]  Iris Vessey,et al.  Cognitive Fit: A Theory‐Based Analysis of the Graphs Versus Tables Literature* , 1991 .

[65]  Roy C. Schmidt,et al.  MANAGING DELPHI SURVEYS USING NONPARAMETRIC STATISTICAL TECHNIQUES , 1997 .

[66]  Oana Velcu,et al.  A review of dashboards in performance management: Implications for design and research , 2012, Int. J. Account. Inf. Syst..

[67]  David B. Kaber,et al.  Usability and Safety in Electronic Medical Records Interface Design , 2015, Hum. Factors.

[68]  Miss A.O. Penney.  (b) , 1974, The New Yale Book of Quotations.

[69]  Jos M G A Schols,et al.  A feedback system to improve the quality of nutritional care. , 2013, Nutrition.

[70]  Iain E. Buchan,et al.  Making Audit Actionable: An Example Algorithm for Blood Pressure Management in Chronic Kidney Disease , 2014, AMIA.

[71]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[72]  Edward J. Mulrow,et al.  The Visual Display of Quantitative Information , 1985, Technometrics.

[73]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[74]  Klaus Krippendorff,et al.  Content Analysis: An Introduction to Its Methodology , 1980 .

[75]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[76]  J. Nielsen.  F-shaped pattern for reading Web content, Jakob Nielsen's Alertbox , 2006 .

[77]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[78]  Elske Ammenwerth,et al.  Usability flaws of medication-related alerting functions: A systematic qualitative review , 2015, J. Biomed. Informatics.

[79]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[80]  J. Mainz.  Defining and classifying clinical indicators for quality improvement. , 2003, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[81]  Geraldine Fitzpatrick,et al.  Dashboards for improving patient care: Review of the literature , 2015, Int. J. Medical Informatics.

[82]  Peter Dalgaard,et al.  R Development Core Team (2010): R: A language and environment for statistical computing , 2010 .

[83]  D. Spiegelhalter,et al.  Principles for Best Practice in Clinical Audit , 2022 .