Measuring the Impact of Online Evidence Retrieval Systems using Critical Incidents & Journey Mapping

Online evidence retrieval systems are one potential tool in supporting evidence-based practice. We have undertaken a program of research to investigate how hospital-based clinicians (doctors, nurses and allied health professionals) use these systems, factors influencing use and their impact on decision-making and health care delivery. A central component of this work has been the development and testing of a broad range of evaluation techniques. This paper provides an overview of the results obtained from three stages of this evaluation and details the results derived from the final stage which sought to test two methods for assessing the integration of an online evidence system and its impact on decision making and patient care. The critical incident and journey mapping techniques were applied. Semi-structured interviews were conducted with 29 clinicians who were experienced users of the online evidence system. Clinicians were asked to described recent instances in which the information obtained using the online evidence system was especially helpful with their work. A grounded approach to data analysis was taken producing three categories of impact. The journey mapping technique was adapted as a method to describe and quantify clinicians' integration of CIAP into their practice and the impact of this on patient care. The analogy of a journey is used to capture the many stages in this integration process, from introduction to the system to full integration into everyday clinical practice with measurable outcomes. Transcribed interview accounts of system use were mapped against the journey stages and scored. Clinicians generated 85 critical incidents and one quarter of these provided specific examples of system use leading to improvements in patient care. The journey mapping technique proved to be a useful method for providing a quantification of the ways and extent to which clincians had integrated system use into practice, and insights into how information systems can influence organisational culture. Further work is required on this technique to assess its value as an evaluation method. The study demonstrates the strength of a triangulated evidence approach to assessing the use and impact of online clinical evidence systems.

[1]  J. Marshall,et al.  The impact of the hospital library on clinical decision making: the Rochester study. , 1992, Bulletin of the Medical Library Association.

[2]  D. N. King,et al.  The contribution of hospital library information services to clinical care: a study in eight hospitals. , 1987, Bulletin of the Medical Library Association.

[3]  K. A. McKibbon,et al.  Online access to MEDLINE in clinical settings. A study of use and usefulness. , 1990, Annals of internal medicine.

[4]  Johanna I. Westbrook,et al.  Research Paper: Clinical Team Functioning and IT Innovation: A Study of the Diffusion of a Point-of-care Online Evidence System , 2003, J. Am. Medical Informatics Assoc..

[5]  M. S. Klein,et al.  Effect of online literature searching on length of stay and patient care costs , 1994, Academic medicine : journal of the Association of American Medical Colleges.

[6]  Norman E. Sondak,et al.  Computers and medicine , 1979 .

[7]  Johanna I. Westbrook,et al.  Research Paper: Do clinicians use online evidence to support patient care? a study of 55, 000 clinicians , 2003, J. Am. Medical Informatics Assoc..

[8]  Johanna I. Westbrook,et al.  Allied health professionals' use of online evidence: a survey of 790 staff working in the Australian public hospital system , 2004, Int. J. Medical Informatics.

[9]  Johanna I Westbrook,et al.  Nurses' use of online clinical evidence. , 2004, Journal of advanced nursing.

[10]  J. C. Flanagan Psychological Bulletin THE CRITICAL INCIDENT TECHNIQUE , 2022 .

[11]  Barry M. Kibel Success Stories as Hard Data: An Introduction to Results Mapping , 1999 .

[12]  A. Strauss,et al.  The Discovery of Grounded Theory , 1967 .

[13]  Johanna I. Westbrook,et al.  Variation in the use of online clinical evidence: a qualitative analysis , 2003, Int. J. Medical Informatics.

[14]  R A Elliott,et al.  The Clinical Information Access Project , 2000, The Medical journal of Australia.

[15]  B. A. Rapp,et al.  Use of MEDLINE by physicians for clinical problem solving. , 1993, JAMA.

[16]  Simon de Lusignan,et al.  National electronic Library for Health (NeLH) , 1999, BMJ.

[17]  J I Westbrook,et al.  Use of point‐of‐care online clinical evidence by junior and senior doctors in New South Wales public hospitals , 2005, Internal medicine journal.

[18]  Linda A. Watson,et al.  Information Retrieval: A Health and Biomedical Perspective. , 2005 .

[19]  W. Hersh Information Retrieval: A Health Care Perspective , 1995, Computers and Medicine.