Developing Software to “Track and Catch” Missed Follow-up of Abnormal Test Results in a Complex Sociotechnical Environment

BACKGROUND Abnormal test results do not always receive timely follow-up, even when providers are notified through electronic health record (EHR)-based alerts. High workload, alert fatigue, and other demands on attention disrupt a provider's prospective memory for tasks required to initiate follow-up. Thus, EHR-based tracking and reminding functionalities are needed to improve follow-up. OBJECTIVES The purpose of this study was to develop a decision-support software prototype enabling individual and system-wide tracking of abnormal test result alerts lacking follow-up, and to conduct formative evaluations, including usability testing. METHODS We developed a working prototype software system, the Alert Watch And Response Engine (AWARE), to detect abnormal test result alerts lacking documented follow-up, and to present context-specific reminders to providers. Development and testing took place within the VA's EHR and focused on four cancer-related abnormal test results. Design concepts emphasized mitigating the effects of high workload and alert fatigue while being minimally intrusive. We conducted a multifaceted formative evaluation of the software, addressing fit within the larger socio-technical system. Evaluations included usability testing with the prototype and interview questions about organizational and workflow factors. Participants included 23 physicians, 9 clinical information technology specialists, and 8 quality/safety managers. RESULTS Evaluation results indicated that our software prototype fit within the technical environment and clinical workflow, and physicians were able to use it successfully. Quality/safety managers reported that the tool would be useful in future quality assurance activities to detect patients who lack documented follow-up. Additionally, we successfully installed the software on the local facility's "test" EHR system, thus demonstrating technical compatibility. CONCLUSION To address the factors involved in missed test results, we developed a software prototype to account for technical, usability, organizational, and workflow needs. Our evaluation has shown the feasibility of the prototype as a means of facilitating better follow-up for cancer-related abnormal test results.

[1]  Edwin Hutchins,et al.  How a Cockpit Remembers Its Speeds , 1995, Cogn. Sci..

[2]  Jonathan Grudin,et al.  Groupware and social dynamics: eight challenges for developers , 1994, CACM.

[3]  Dean F Sittig,et al.  Electronic health record-based messages to primary care providers: valuable information or just noise? , 2012, Archives of internal medicine.

[4]  Alan Hedge,et al.  Cognitive ergonomics, socio-technical systems, and the impact of healthcare information technologies , 2011 .

[5]  David D. Woods,et al.  Systems with Human Monitors: A Signal Detection Analysis , 1985, Hum. Comput. Interact..

[6]  Stephen L. Hillis,et al.  Prevalence of delayed clinician response to elevated prostate-specific antigen values. , 2008, Mayo Clinic proceedings.

[7]  George Dunea,et al.  A medical error , 2000, BMJ : British Medical Journal.

[8]  Vimla L. Patel,et al.  The multitasking clinician: Decision-making and cognitive demand during and after team handoffs in emergency care , 2007, Int. J. Medical Informatics.

[9]  N. Franklin,et al.  Diagnostic error in internal medicine. , 2005, Archives of internal medicine.

[10]  Hardeep Singh,et al.  Reducing Diagnostic Errors through Effective Communication: Harnessing the Power of Information Technology , 2008, Journal of General Internal Medicine.

[11]  Christopher Nemeth,et al.  The Messy Details: Insights From the Study of Technical Work in Healthcare , 2004 .

[12]  R. Wears,et al.  Computer technology and clinical work: still waiting for Godot. , 2005, JAMA.

[13]  W. Dunsmuir,et al.  The impact of interruptions on clinical task completion , 2010, Quality and Safety in Health Care.

[14]  Hardeep Singh,et al.  Research paper: Provider management strategies of abnormal test result alerts: a cognitive task analysis , 2010, J. Am. Medical Informatics Assoc..

[15]  Monique W. M. Jaspers,et al.  A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence , 2009, Int. J. Medical Informatics.

[16]  Emily S. Patterson,et al.  Exploring barriers and facilitators to the use of computerized clinical reminders. , 2005, Journal of the American Medical Informatics Association : JAMIA.

[17]  Hardeep Singh,et al.  Improving follow-up of abnormal cancer screens using electronic health records: trust but verify test result communication , 2009, BMC Medical Informatics Decis. Mak..

[18]  Traber Davis,et al.  Characteristics and predictors of missed opportunities in lung cancer diagnosis: an electronic health record-based study. , 2010, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[19]  Penelope M. Sanderson,et al.  Distributed Prospective Memory: An Approach to Understanding how Nurses Remember Tasks , 2009 .

[20]  Alissa L. Russ,et al.  Current Challenges and Opportunities for Better Integration of Human Factors Research with Development of Clinical Information Systems , 2009, Yearbook of Medical Informatics.

[21]  Hardeep Singh,et al.  Errors in cancer diagnosis: current understanding and future directions. , 2007, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[22]  Terry Wahls,et al.  Diagnostic Errors and Abnormal Diagnostic Tests Lost to Follow‐Up: A Source of Needless Waste and Delay to Treatment , 2007, The Journal of ambulatory care management.

[23]  Eric G. Poon,et al.  Research Paper: The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry , 2007, J. Am. Medical Informatics Assoc..

[24]  Dean F Sittig,et al.  A new sociotechnical model for studying health information technology in complex adaptive healthcare systems , 2010, Quality and Safety in Health Care.

[25]  Clement J. McDonald,et al.  Electronic medical records and preserving primary care physicians' time: comment on "electronic health record-based messages to primary care providers". , 2012, Archives of Internal Medicine.

[26]  Hardeep Singh,et al.  Using a Multifaceted Approach to Improve the Follow-Up of Positive Fecal Occult Blood Test Results , 2009, The American Journal of Gastroenterology.

[27]  J. Carroll,et al.  Moving Beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems , 2009 .

[28]  Dean F Sittig,et al.  Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. , 2012, The American journal of medicine.

[29]  Robert L. Wears,et al.  Health information technology: fallacies and sober realities , 2010, J. Am. Medical Informatics Assoc..

[30]  P. Carayon,et al.  Work system design for patient safety: the SEIPS model , 2006, Quality and Safety in Health Care.

[31]  Hardeep Singh,et al.  Eight recommendations for policies for communicating abnormal test results. , 2010, Joint Commission journal on quality and patient safety.

[32]  Hardeep Singh,et al.  Research Paper: Communication Outcomes of Critical Imaging Results in a Computerized Notification System , 2007, J. Am. Medical Informatics Assoc..

[33]  Joseph S. Dumas,et al.  Usability assessment methods , 2006 .

[34]  Dean F Sittig,et al.  Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? , 2010, The American journal of medicine.

[35]  Dean F. Sittig,et al.  Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? , 2009, Archives of internal medicine.

[36]  J. Reason,et al.  Combating omission errors through task analysis and good reminders , 2002, Quality & safety in health care.

[37]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[38]  Michael R Hamblin,et al.  CA : A Cancer Journal for Clinicians , 2011 .

[39]  Stephen S Raab,et al.  Quality in Cancer Diagnosis , 2010, CA: a cancer journal for clinicians.

[40]  David Woods,et al.  How Not to Have to Navigate Through Too Many Displays , 1997 .

[41]  Hardeep Singh,et al.  Exploring situational awareness in diagnostic errors in primary care , 2011, BMJ quality & safety.

[42]  R. Key Dismukes,et al.  Remembrance of Things Future: Prospective Memory in Laboratory, Workplace, and Everyday Settings , 2010 .

[43]  P. Carayon,et al.  Sociotechnical systems analysis in health care: a research agenda , 2011, IIE transactions on healthcare systems engineering.

[44]  Philip J. Smith,et al.  Brittleness in the design of cooperative problem-solving systems: the effects on user performance , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[45]  Hardeep Singh,et al.  Reducing diagnostic error through medical home-based primary care reform. , 2010, JAMA.

[46]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[47]  Brian P. Bailey,et al.  If not now, when?: the effects of interruption at different moments within task execution , 2004, CHI.

[48]  Steven B. Zeliadt,et al.  What Happens After an Elevated PSA Test: The Experience of 13,591 Veterans , 2010, Journal of General Internal Medicine.

[49]  Dean F Sittig,et al.  Information overload and missed test results in electronic health record-based settings. , 2013, JAMA internal medicine.

[50]  Stephen J. Morrissey,et al.  Reviews of Human Factors and Ergonomics , 2008 .

[51]  Gordon D Schiff,et al.  Medical error: a 60-year-old man with delayed care for a renal mass. , 2011, JAMA.

[52]  David W Bates,et al.  "I wish I had seen this test result earlier!": Dissatisfaction with test result management systems in primary care. , 2004, Archives of internal medicine.

[53]  Waldemar Karwowski,et al.  Applying Cognitive Psychology to System Development , 2006 .

[54]  Michael W. Smith,et al.  Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey , 2012, J. Am. Medical Informatics Assoc..

[55]  Ellen J. Bass,et al.  Handoff Communication: Implications For Design , 2012 .

[56]  Simran Singh,et al.  Understanding the management of electronic test result notifications in the outpatient setting , 2011, BMC Medical Informatics Decis. Mak..

[57]  T. Brennan,et al.  Missed and Delayed Diagnoses in the Ambulatory Setting: A Study of Closed Malpractice Claims , 2006, Annals of Internal Medicine.

[58]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[59]  David W. Bates,et al.  Design and implementation of a comprehensive outpatient Results Manager , 2003, J. Biomed. Informatics.

[60]  Stephanie Guerlain,et al.  Interactive Critiquing as a Form of Decision Support: An Empirical Evaluation , 1999, Hum. Factors.

[61]  Marc Berg,et al.  Review Paper: Overriding of Drug Safety Alerts in Computerized Physician Order Entry , 2006, J. Am. Medical Informatics Assoc..

[62]  Dean F. Sittig,et al.  Ten Strategies to Improve Management of Abnormal Test Result Alerts in the Electronic Health Record , 2010, Journal of patient safety.