Complementary methods of system usability evaluation: Surveys and observations during software design and development cycles

Poor usability of clinical information systems delays their adoption by clinicians and limits potential improvements to the efficiency and safety of care. Recurring usability evaluations are therefore, integral to the system design process. We compared four methods employed during the development of outpatient clinical documentation software: clinician email response, online survey, observations and interviews. Results suggest that no single method identifies all or most problems. Rather, each approach is optimal for evaluations at a different stage of design and characterizes different usability aspect. Email responses elicited from clinicians and surveys report mostly technical, biomedical, terminology and control problems and are most effective when a working prototype has been completed. Observations of clinical work and interviews inform conceptual and workflow-related problems and are best performed early in the cycle. Appropriate use of these methods consistently during development may significantly improve system usability and contribute to higher adoption rates among clinicians and to improved quality of care.

[1]  John Doucette,et al.  Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries , 2009, Int. J. Medical Informatics.

[2]  Beth F. Wheeler Atkinson,et al.  Development of a Multiple Heuristics Evaluation Table (MHET) to Support Software Development and Usability Analysis , 2007, HCI.

[3]  Suzanne Bakken,et al.  Heuristic evaluation of a Web-based Educational Resource for low literacy NICU parents. , 2006, Studies in health technology and informatics.

[4]  Kasper Hornbæk,et al.  Tracing impact in a usability improvement process , 2008, Interact. Comput..

[5]  J V Douglas,et al.  Failure to provide clinicians useful IT systems: opportunities to leapfrog current technologies. , 2008, Methods of information in medicine.

[6]  Tsai-Ya Lai.  Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing , 2007, Int. J. Medical Informatics.

[7]  Jennifer Lai,et al.  Unintended Consequences of Information Technologies in Health Care—An Interactive Sociotechnical Analysis , 2007 .

[8]  Vimla L. Patel,et al.  Heuristic evaluation of paper-based Web pages: A simplified inspection usability methodology , 2006, J. Biomed. Informatics.

[9]  Gilad J. Kuperman,et al.  Case Report: Comprehensive Analysis of a Medication Dosing Error Related to CPOE , 2005, J. Am. Medical Informatics Assoc..

[10]  Clayton Lewis,et al.  Designing for usability—key principles and what designers think , 1983, CHI '83.

[11]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[12]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[13]  Eric W. Ford,et al.  Predicting computerized physician order entry system adoption in US hospitals: Can the federal mandate be met? , 2008, Int. J. Medical Informatics.

[14]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[15]  Dongwen Wang,et al.  Using multi-perspective methodologies to study users' interactions with the prototype front end of a guideline-based decision support system for diabetic foot care , 2009, Int. J. Medical Informatics.

[16]  Steven R. Simon,et al.  Research Paper: Physicians' Use of Key Functions in Electronic Health Records from 2005 to 2007: A Statewide Survey , 2009, J. Am. Medical Informatics Assoc..

[17]  Qi Li,et al.  "Smart Forms" in an Electronic Medical Record: documentation-based clinical decision support to improve disease management. , 2008, Journal of the American Medical Informatics Association : JAMIA.

[18]  D. Bates,et al.  Electronic health record use and the quality of ambulatory care in the United States. , 2007, Archives of internal medicine.

[19]  B. Marx.  The Visual Display of Quantitative Information , 1985 .

[20]  S. Koch,et al.  Heuristic evaluation performed by usability-educated clinicians: education and attitudes. , 2007, Studies in health technology and informatics.

[21]  ELSKE AMMENWERTH,et al.  Review Paper: The Effect of Electronic Prescribing on Medication Errors and Adverse Drug Events: A Systematic Review , 2008, J. Am. Medical Informatics Assoc..

[22]  Valerie L. Shalin,et al.  Cognitive task analysis , 2000 .

[23]  Frank Drews,et al.  Research Paper: The Employment of an Iterative Design Process to Develop a Pulmonary Graphical Display , 2003, J. Am. Medical Informatics Assoc..

[24]  William W. Stead,et al.  Review: Computer-based Physician Order Entry: The State of the Art , 1994, J. Am. Medical Informatics Assoc..

[25]  Vimla L. Patel,et al.  Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units , 2004, Int. J. Medical Informatics.

[26]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[27]  Sowmya R. Rao,et al.  Electronic health records in ambulatory care--a national survey of physicians. , 2008, The New England journal of medicine.

[28]  A. Localio,et al.  Role of computerized physician order entry systems in facilitating medication errors. , 2005, JAMA.

[29]  Kai Zheng,et al.  Research Article: An Interface-driven Analysis of User Interactions with an Electronic Health Records System , 2009, J. Am. Medical Informatics Assoc..

[30]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th Edition) , 2004 .

[31]  N. Schwarz,et al.  Asking Questions About Behavior: Cognition, Communication, and Questionnaire Construction , 2001 .

[32]  Himss Ehr Usability Task Force,et al.  Defining and Testing EMR Usability , 2009 .

[33]  Jiajie Zhang,et al.  UFuRT: A Work ⴀ Centered Framework and Process for Design and Evaluation of Information Systems , 2007 .

[34]  John B. Smelcer,et al.  Usability of electronic medical records , 2009 .

[35]  Juliet M. Corbin,et al.  Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory , 2008 .

[36]  James P. Turley,et al.  The significance of cognitive modeling in building healthcare interfaces , 2006, Int. J. Medical Informatics.

[37]  Bonnie Kaplan,et al.  White Paper: Health IT Success and Failure: Recommendations from Literature and an AMIA Workshop , 2009, J. Am. Medical Informatics Assoc..

[38]  Titus K L Schleyer,et al.  Heuristic evaluation of clinical functions in four practice management systems: a pilot study. , 2007, Journal of the American Dental Association.

[39]  Bruce Tognazzini,et al.  Tog on Interface , 1992 .

[40]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[41]  Jeffery L. Belden,et al.  Defining and testing EMR usability: Principles and proposed methods of EMR usability evaluation and rating. , 2009 .

[42]  Nancy G. Leveson,et al.  Viewpoint Paper: EHR Safety: The Way Forward to Safe and Effective Systems , 2008, J. Am. Medical Informatics Assoc..

[43]  Andrés Silva,et al.  A Conceptual Model for the Analysis of Mishaps in Human-Operated Safety-Critical Systems , 2008 .

[44]  Jakob Nielsen,et al.  Iterative user-interface design , 1993, Computer.

[45]  Charlene R. Weir,et al.  Research Paper: A Cognitive Task Analysis of Information Management Strategies in a Computerized Provider Order Entry Environment , 2007, J. Am. Medical Informatics Assoc..

[46]  Dean F Sittig,et al.  Eight rights of safe electronic health record use. , 2009, JAMA.

[47]  D. Bates,et al.  Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. , 2003, Archives of internal medicine.

[48]  Nancy G. Leveson.  Intent Specifications: An Approach to Building Human-Centered Specifications , 2000, IEEE Trans. Software Eng..

[49]  Jiajie Zhang,et al.  Applying heuristic evaluation to improve the usability of a telemedicine system. , 2006, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[50]  B. Schneirdeman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[51]  P. Fayers,et al.  The Visual Display of Quantitative Information , 1990 .

[52]  David W. Bates,et al.  Primary care physician time utilization before and after implementation of an electronic health record: A time-motion study , 2005, J. Biomed. Informatics.

[53]  P. Shekelle,et al.  Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care , 2006, Annals of Internal Medicine.