Methodological Review Cognitive and usability engineering methods for the evaluation of clinical information systems

Increasingly healthcare policy and decision makers are demanding evidence to justify investments in health information systems. This demand requires an adequate evaluation of these systems. A wide variety of approaches and methodologies have been applied in assessing the impact of information systems in health care, ranging from controlled clinical trials to use of questionnaires and interviews with users. In this paper we describe methodological approaches which we have applied and refined for the past 10 years for the evaluation of health information systems. The approaches are strongly rooted in theories and methods from cognitive science and the emerging field of usability engineering. The focus is on assessing human computer interaction and in particular, the usability of computer systems in both laboratory and naturalistic settings. The methods described can be a part of the formative evaluation of systems during their iterative development, and can also complement more traditional assessment methods used in summative system evaluation of completed systems. The paper provides a review of the general area of systems evaluation with the motivation and rationale for methodological approaches underlying usability engineering and cognitive task analysis as applied to health information systems. This is followed by a detailed description of the methods we have applied in a variety of settings in conducting usability testing and usability inspection of systems such as computer-based patient records. Emerging trends in the evaluation of complex information systems are discussed.

[1]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[2]  K. A. Ericsson,et al.  Toward a general theory of expertise : prospects and limits , 1991 .

[3]  Vimla L. Patel,et al.  'Televaluation' of clinical information systems: an integrative approach to assessing Web-based systems , 2001, Int. J. Medical Informatics.

[4]  Vimla L. Patel,et al.  Review: A Primer on Aspects of Cognition for Medical Informatics , 2001, J. Am. Medical Informatics Assoc..

[5]  J Brender,et al.  Cognitive evaluation: how to assess the usability of information technology in healthcare. , 1997, Computer methods and programs in biomedicine.

[6]  Marie-Catherine Beuscart-Zéphir,et al.  A rapid usability assessment methodology to support the choice of clinical information systems: a case study , 2002, AMIA.

[7]  Keith Duncan,et al.  Cognitive Engineering , 2017, Encyclopedia of GIS.

[8]  Vimla L. Patel,et al.  Designing Human-Centered Distributed Information Systems , 2002, IEEE Intell. Syst..

[9]  Tom Carey,et al.  Human-computer interaction , 1994 .

[10]  Barbara Means,et al.  Cognitive task analysis as a basis for tutor development: Articulating abstract knowledge representations. , 1988 .

[11]  Jakob Nielsen,et al.  Designing web usability , 1999 .

[12]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[13]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[14]  Jens Rasmussen,et al.  Cognitive Systems Engineering , 2022 .

[15]  K. J. Vicente,et al.  Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work , 1999 .

[16]  Christian Köhler,et al.  How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews , 2002, BMJ : British Medical Journal.

[17]  Sri Hastuti Kurniawan,et al.  Review of Interaction design , 2003 .

[18]  Stephanie Rosenbaum,et al.  Usability in practice , 2002, CHI 2002.

[19]  Vimla L. Patel,et al.  Cognitive evaluation of decision making processes and assessment of information technology in medicine , 1998, Int. J. Medical Informatics.

[20]  André Kushniruk,et al.  Analysis of Complex Decision-Making Processes in Health Care: Cognitive Approaches to Health Informatics , 2001, J. Biomed. Informatics.

[21]  De Klerk,et al.  Web site usability , 2001 .

[22]  Marilyn Hughes Blackmon,et al.  A Comprehension-based Model of Web Navigation and Its Application to Web Usability Analysis , 2000, BCS HCI.

[23]  D. Norman.  Categorization of action slips. , 1981 .

[24]  V. Patel,et al.  Assessment of a computerized patient record system: a cognitive approach to evaluating medical technology. , 1996, M.D. computing : computers in medical practice.

[25]  Steve McConnell,et al.  Rapid Development: Taming Wild Software Schedules , 1996 .

[26]  Vimla L. Patel,et al.  Emerging paradigms of cognition in medical decision-making , 2002, J. Biomed. Informatics.

[27]  Vimla L. Patel,et al.  A framework for analyzing the cognitive complexity of computer-assisted clinical ordering , 2003, J. Biomed. Informatics.

[28]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[29]  Vimla L. Patel,et al.  Research Paper: Impact of a Computer-based Patient Record System on Data Collection, Knowledge Organization, and Reasoning , 2000, J. Am. Medical Informatics Assoc..

[30]  Randolph G. Bias,et al.  The pluralistic usability walkthrough: coordinated empathies , 1994 .

[31]  Joseph Psotka,et al.  Intelligent tutoring systems : lessons learned , 1988 .

[32]  Vimla L. Patel,et al.  Research Paper: Studying the Human- Computer-Terminology Interface , 2001, J. Am. Medical Informatics Assoc..

[33]  André Kushniruk,et al.  Evaluation in the design of health information systems: application of approaches emerging from usability engineering , 2002, Comput. Biol. Medicine.

[34]  Peter D. Stetson,et al.  Application of Information Technology: PalmCIS: A Wireless Handheld Application for Satisfying Clinician Information Needs , 2004, J. Am. Medical Informatics Assoc..

[35]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[36]  John M. Carroll,et al.  Human-Computer Interaction in the New Millennium , 2001 .

[37]  Scott Weiss.  Handheld usability , 2002, DIS '02.

[38]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[39]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[40]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[41]  Vimla L. Patel,et al.  Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces , 1997, AMIA.

[42]  Charles P. Friedman,et al.  Evaluation and Technology Assessment , 2001 .

[43]  C. Aydin,et al.  Evaluating Health Care Information Systems: Methods and Applications , 1993 .

[44]  Colin F. Mackenzie,et al.  SPECIAL SECTION: Comparison of Self-Reporting of Deficiencies in Airway Management with Video Analyses of Actual Performance , 1996, Hum. Factors.

[45]  Cathleen Wharton,et al.  The cognitive walkthrough method: a practitioner's guide , 1994 .

[46]  Jens Rasmussen,et al.  Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering , 1986 .

[47]  A M Wichansky,et al.  Usability testing in 2000 and beyond , 2000, Ergonomics.

[48]  V L Patel,et al.  Cognitive computer-based video analysis: its application in assessing the usability of medical systems. , 1995, Medinfo. MEDINFO.

[49]  John Karat,et al.  Iterative usability testing: ensuring a usable clinical workstation , 1997, AMIA.

[50]  Christopher G. Chute,et al.  Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System , 1997, AMIA.

[51]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[52]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[53]  J J Cimino,et al.  Cognitive evaluation of the user interface and vocabulary of an outpatient information system. , 1996, Proceedings : a conference of the American Medical Informatics Association. AMIA Fall Symposium.

[54]  Vimla L. Patel,et al.  Usability in the real world: assessing medical information technologies in patients' homes , 2003, J. Biomed. Informatics.

[55]  Vimla L. Patel,et al.  Usability evaluation of an experimental text summarization system and three search engines: implications for the reengineering of health care interfaces , 2002, AMIA.

[56]  Mark Pearrow.  The Wireless Web Usability Handbook , 2002 .

[57]  J. Fox,et al.  Decision-Support Systems as Safety-Critical Components: Towards a Safety Culture for Medical Informatics , 1993, Methods of Information in Medicine.