Cognitive Approaches to the Evaluation of Healthcare Information Systems

This chapter provides an overview of cognitive approaches to the evaluation of healthcare information systems. Cognitive approaches in health informatics focus on understanding the processes involved in the decision making and reasoning of healthcare workers as they interact with information systems to carry out a range of tasks. In the first part of the chapter the motivation and theoretical background to cognitive evaluation are provided. The importance of developing effective methods for understanding how systems impact on cognitive processes is discussed as well as the need for developing new approaches to system evaluation borrowing from advances in cognitive science and the study of human–computer interaction. In particular, methods emerging from the areas of usability engineering and cognitive task analysis have important implications for the improved assessment of cognition involved in complex medical tasks and the impact of information systems. Methodologies are described for considering evaluation throughout the system design and development life cycle. The chapter then illustrates how research in cognitive science can be used to drive the development of new conceptual frameworks for evaluation of healthcare information systems. Specific examples from our research will be provided, ranging from application of cognitive approaches for the laboratory analysis of user interactions with complex information systems such as electronic medical records, to the cognitive evaluation of Web-based information resources. A wide variety of approaches have been taken in the evaluation of healthcare information systems. Many of these evaluations have focused on assessing outcomes associated with deployment and use of systems in clinical environments. These studies have typically involved measurement of dependent variables such as cost of health care, quality of care, and other outcomes [1]. Although such summative evaluation of completed healthcare information systems is necessary to ensure their effectiveness, in recent years an increasing emphasis has appeared on the in-depth study of the 6 Cognitive Approaches to the Evaluation of Healthcare Information Systems

[1]  Vimla L. Patel,et al.  Research Paper: Studying the Human- Computer-Terminology Interface , 2001, J. Am. Medical Informatics Assoc..

[2]  Valerie L. Shalin,et al.  Cognitive task analysis , 2000 .

[3]  Vimla L. Patel,et al.  Cognitive evaluation of decision making processes and assessment of information technology in medicine , 1998, Int. J. Medical Informatics.

[4]  André Kushniruk,et al.  Analysis of Complex Decision-Making Processes in Health Care: Cognitive Approaches to Health Informatics , 2001, J. Biomed. Informatics.

[5]  J J Cimino,et al.  Cognitive evaluation of the user interface and vocabulary of an outpatient information system. , 1996, Proceedings : a conference of the American Medical Informatics Association. AMIA Fall Symposium.

[6]  V L Patel,et al.  Cognitive computer-based video analysis: its application in assessing the usability of medical systems. , 1995, Medinfo. MEDINFO.

[7]  John Karat,et al.  Iterative usability testing: ensuring a usable clinical workstation , 1997, AMIA.

[8]  Vimla L. Patel,et al.  Usability testing in medical informatics: cognitive approaches to evaluation of information systems and user interfaces , 1997, AMIA.

[9]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[10]  K. J. Vicente,et al.  Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work , 1999 .

[11]  Jochen R. Moehr.  Evaluation: salvation or nemesis of medical informatics? , 2002, Comput. Biol. Medicine.

[12]  Vimla L. Patel,et al.  The patient clinical information system (PatCIS): technical solutions for and experience with giving patients access to their electronic medical records , 2002, Int. J. Medical Informatics.

[13]  Steve McConnell,et al.  Rapid Development: Taming Wild Software Schedules , 1996 .

[14]  Vimla L. Patel,et al.  Diagnostic Reasoning and Medical Expertise , 1994 .

[15]  K. A. Ericsson,et al.  Toward a general theory of expertise : prospects and limits , 1991 .

[16]  Jens Rasmussen,et al.  Cognitive Systems Engineering , 2022 .

[17]  Joseph Psotka,et al.  Intelligent tutoring systems : lessons learned , 1988 .

[18]  Vimla L. Patel,et al.  Usability in the real world: assessing medical information technologies in patients' homes , 2003, J. Biomed. Informatics.

[19]  Vimla L. Patel,et al.  Cognitive and usability engineering methods for the evaluation of clinical information systems , 2004, J. Biomed. Informatics.

[20]  André Kushniruk,et al.  Evaluation in the design of health information systems: application of approaches emerging from usability engineering , 2002, Comput. Biol. Medicine.

[21]  Robert M. Hamm,et al.  Moment-by-moment variation in experts' analytic and intuitive cognitive activity , 1988, IEEE Trans. Syst. Man Cybern..

[22]  Christopher G. Chute,et al.  Standardized problem list generation, utilizing the Mayo canonical vocabulary embedded within the Unified Medical Language System , 1997, AMIA.

[23]  Jeffrey Rubin,et al.  Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests , 1994 .

[24]  Vimla L. Patel,et al.  'Televaluation' of clinical information systems: an integrative approach to assessing Web-based systems , 2001, Int. J. Medical Informatics.

[25]  Stephanie Rosenbaum,et al.  Usability in practice , 2002, CHI 2002.

[26]  V. Patel,et al.  Assessment of a computerized patient record system: a cognitive approach to evaluating medical technology. , 1996, M.D. computing : computers in medical practice.

[27]  G. Klein,et al.  Decision Making in Action: Models and Methods , 1993 .

[28]  Barbara Means,et al.  Cognitive task analysis as a basis for tutor development: Articulating abstract knowledge representations. , 1988 .

[29]  John M. Carroll,et al.  Human-Computer Interaction in the New Millennium , 2001 .

[30]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[31]  Yvonne Rogers,et al.  Interaction Design: Beyond Human-Computer Interaction , 2002 .

[32]  Elizabeth M Borycki,et al.  Technology induced error and usability: The relationship between usability problems and prescription errors when using a handheld application , 2005, Int. J. Medical Informatics.

[33]  Vimla L. Patel,et al.  A framework for analyzing the cognitive complexity of computer-assisted clinical ordering , 2003, J. Biomed. Informatics.

[34]  Vimla L. Patel,et al.  Usability evaluation of an experimental text summarization system and three search engines: implications for the reengineering of health care interfaces , 2002, AMIA.

[35]  André Kushniruk,et al.  Bringing usability to the early stages of software development , 2003, Proceedings. 11th IEEE International Requirements Engineering Conference, 2003..

[36]  David W. Bates,et al.  Cognition and measurement in patient safety research , 2003, J. Biomed. Informatics.

[37]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[38]  Ben Shneiderman,et al.  Designing The User Interface , 2013 .

[39]  G. Klein,et al.  A recognition-primed decision (RPD) model of rapid decision making. , 1993 .

[40]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[41]  Vimla L. Patel,et al.  Research Paper: Impact of a Computer-based Patient Record System on Data Collection, Knowledge Organization, and Reasoning , 2000, J. Am. Medical Informatics Assoc..