Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study

BackgroundComputerized decision support systems (DSS) have mainly focused on improving clinicians' diagnostic accuracy in unusual and challenging cases. However, since diagnostic omission errors may predominantly result from incomplete workup in routine clinical practice, the provision of appropriate patient- and context-specific reminders may result in greater impact on patient safety. In this experimental study, a mix of easy and difficult simulated cases were used to assess the impact of a novel diagnostic reminder system (ISABEL) on the quality of clinical decisions made by various grades of clinicians during acute assessment.MethodsSubjects of different grades (consultants, registrars, senior house officers and medical students), assessed a balanced set of 24 simulated cases on a trial website. Subjects recorded their clinical decisions for the cases (differential diagnosis, test-ordering and treatment), before and after system consultation. A panel of two pediatric consultants independently provided gold standard responses for each case, against which subjects' quality of decisions was measured. The primary outcome measure was change in the count of diagnostic errors of omission (DEO). A more sensitive assessment of the system's impact was achieved using specific quality scores; additional consultation time resulting from DSS use was also calculated.Results76 subjects (18 consultants, 24 registrars, 19 senior house officers and 15 students) completed a total of 751 case episodes. The mean count of DEO fell from 5.5 to 5.0 across all subjects (repeated measures ANOVA, p < 0.001); no significant interaction was seen with subject grade. Mean diagnostic quality score increased after system consultation (0.044; 95% confidence interval 0.032, 0.054). ISABEL reminded subjects to consider at least one clinically important diagnosis in 1 in 8 case episodes, and prompted them to order an important test in 1 in 10 case episodes. Median extra time taken for DSS consultation was 1 min (IQR: 30 sec to 2 min).ConclusionThe provision of patient- and context-specific reminders has the potential to reduce diagnostic omissions across all subject grades for a range of cases. This study suggests a promising role for the use of future reminder-based DSS in the reduction of diagnostic error.

[1]  W W Tunnessen,et al.  Computer-assisted diagnosis in pediatrics. , 1974, American journal of diseases of children.

[2]  Randolph A. Miller,et al.  Research Paper: Evaluation of Long-term Maintenance of a Large Medical Knowledge Base , 1995, J. Am. Medical Informatics Assoc..

[3]  Randolph A. Miller,et al.  Evaluating Evaluations of Medical Diagnostic Systems , 1996, J. Am. Medical Informatics Assoc..

[4]  D. Bates,et al.  Information technology and medication safety: what is the benefit? , 2002, Quality & safety in health care.

[5]  ETA S. BERNER,et al.  Review Paper: Will the Wave Finally Break? A Brief View of the Adoption of Electronic Medical Records in the United States , 2004, J. Am. Medical Informatics Assoc..

[6]  T. Brennan,et al.  Incidence and types of adverse events and negligent care in Utah and Colorado. , 2000, Medical care.

[7]  C. McDonald,et al.  A computerized reminder system to increase the use of preventive care for hospitalized patients. , 2001, The New England journal of medicine.

[8]  S A Spooner,et al.  Medical informatics and pediatrics. , 1996, Archives of pediatrics & adolescent medicine.

[9]  P Ramnarayan,et al.  Paediatric clinical decision support systems , 2002, Archives of disease in childhood.

[10]  T YESHURUN-BERMAN,et al.  [The problems of general practice]. , 1962, Harefuah.

[11]  Karen Cosby,et al.  Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project , 2005 .

[12]  Jane Garbutt,et al.  Patient concerns about medical errors in emergency departments. , 2005, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[13]  N. Franklin,et al.  Diagnostic error in internal medicine. , 2005, Archives of internal medicine.

[14]  L. Kohn,et al.  To Err Is Human : Building a Safer Health System , 2007 .

[15]  Brian E Clauser,et al.  The Introduction of Computer‐based Case Simulations into the United States Medical Licensing Examination , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[16]  Jeremy Wyatt,et al.  Decision Aids and the Law , 1989, The Lancet.

[17]  Padmanabhan Ramnarayan,et al.  A novel diagnostic aid (ISABEL): development and preliminary evaluation of clinical performance , 2004, MedInfo.

[18]  R. Gibberd,et al.  An analysis of the causes of adverse events from the Quality in Australian Health Care Study , 1999, The Medical journal of Australia.

[19]  H R Warner.  Iliad: moving medical decision-making into new frontiers. , 1989, Methods of information in medicine.

[20]  M. J. Feldman,et al.  Medical informatics and pediatrics. Decision-support systems. , 1995, Archives of pediatrics & adolescent medicine.

[21]  S. Tanenbaum,et al.  Knowing and acting in medical practice: the epistemological politics of outcomes research. , 1994, Journal of health politics, policy and law.

[22]  P. Davis,et al.  Adverse events in New Zealand public hospitals II: preventability and clinical context. , 2003, The New Zealand medical journal.

[23]  Nancy Franklin,et al.  Reducing Diagnostic Errors in Medicine: What's the Goal? , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[24]  Saxon Ridley.  Critical care. , 2003, Anaesthesia.

[25]  M A Graber,et al.  How well does decision support software perform in the emergency department? , 2003, Emergency medicine journal : EMJ.

[26]  G. K. H. Hodgkin.  Problems in General Practice , 1963 .

[27]  James J. Cimino.  Clinical Knowledge and Practice in the Information Age: A Handbook for Health Professionals , 2001, J. Biomed. Informatics.

[28]  G. Guyatt,et al.  Users' guides to the medical literature. , 1993, JAMA.

[29]  E S Berner,et al.  Influence of case and physician characteristics on perceptions of decision support systems. , 1999, Journal of the American Medical Informatics Association : JAMIA.

[30]  Anne Greenough.  Help from ISABEL for paediatric diagnoses , 2002, The Lancet.

[31]  P. Croskerry.  Achieving quality in clinical decision making: cognitive strategies and detection of bias. , 2002, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[32]  P. D. Wilson,et al.  Diagnosis of acute abdominal pain in the accident and emergency department , 1977, The British journal of surgery.

[33]  Brian E Clauser,et al.  An Examination of the Contribution of Computer‐based Case Simulations to the USMLE Step 3 Examination , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[34]  G O Barnett,et al.  An approach to evaluating the accuracy of DXplain. , 1990, Computer methods and programs in biomedicine.

[35]  A. L. Baker,et al.  Performance of four computer-based diagnostic systems. , 1994, The New England journal of medicine.

[36]  Guido G. Gatti,et al.  Do physicians know when their diagnoses are correct? , 2005, Journal of General Internal Medicine.

[37]  J J Cimino,et al.  Integrating DXplain into a clinical information system using the World Wide Web. , 1996, Proceedings : a conference of the American Medical Informatics Association. AMIA Fall Symposium.

[38]  C R Welford.  A comprehensive computerized patient record with automated linkage to QMR. , 1994, Proceedings. Symposium on Computer Applications in Medical Care.

[39]  D. Bates,et al.  The Critical Care Safety Study: The incidence and nature of adverse events and serious medical errors in intensive care* , 2005, Critical care medicine.

[40]  Randolph A. Miller,et al.  Review: Medical Diagnostic Decision Support Systems - Past, Present, And Future: A Threaded Bibliography and Brief Commentary , 1994, J. Am. Medical Informatics Assoc..

[41]  M S Blois,et al.  Evaluating RECONSIDER. A computer program for diagnostic prompting. , 1985, Journal of medical systems.

[42]  JRA Azevedo,et al.  The impact of nutritional support on morbimortality of critically ill patients , 2003, Critical Care.

[43]  Richard Smith.  What clinical information do doctors need? , 1996, BMJ.

[44]  Randolph A. Miller,et al.  User Variability in Abstracting and Entering Printed Case Histories with QUICK MEDICAL REFERENCE (QMR). , 1987 .

[45]  Eta S. Berner,et al.  Comparison of measures to assess change in diagnostic performance due to a decision support system , 2000, AMIA.

[46]  J Wyatt,et al.  Quantitative evaluation of clinical software, exemplified by decision support systems. , 1997, International journal of medical informatics.

[47]  Eta S. Berner.  Editorial Comments: Diagnostic Decision Support Systems: How to Determine the Gold Standard? , 2003, J. Am. Medical Informatics Assoc..

[48]  W. Hendee.  To Err is Human: Building a Safer Health System , 2001 .

[49]  G H Guyatt,et al.  Users' Guides to the Medical Literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. , 1999, JAMA.

[50]  Gwenifer C M Wilson,et al.  Early days of modern surgery , 1999 .

[51]  F E Masarie,et al.  Quick medical reference (QMR) for diagnostic assistance. , 1986, M.D.Computing.

[52]  Silvia Mamede,et al.  The structure of reflective practice in medicine , 2004, Medical education.

[53]  Neal J Thomas.  Isabel , 2002, Critical Care.

[54]  G. Barnett,et al.  DXplain. An evolving diagnostic decision-support system. , 1987, JAMA.

[55]  R A Miller,et al.  A tool for the computer-assisted creation of QMR medical knowledge base disease profiles. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[56]  Helen Fisher,et al.  ISABEL: support with clinical decision making. , 2003, Paediatric nursing.

[57]  A. Wall,et al.  Book ReviewTo Err is Human: building a safer health system Kohn L T Corrigan J M Donaldson M S Washington DC USA: Institute of Medicine/National Academy Press ISBN 0 309 06837 1 $34.95 , 2000 .

[58]  I. Adams,et al.  Computer aided diagnosis of acute abdominal pain: a multicentre study. , 1986, British medical journal.

[59]  Paul M. Taylor,et al.  Research Paper: Measuring the Impact of Diagnostic Decision Support on the Quality of Clinical Decision Making: Development of a Reliable and Valid Composite Score , 2003, J. Am. Medical Informatics Assoc..

[60]  E. Bartlett,et al.  Physicians' cognitive errors and their liability consequences. , 1998, Journal of healthcare risk management : the journal of the American Society for Healthcare Risk Management.

[61]  Timothy M. Franz,et al.  Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. , 1999, JAMA.

[62]  W. McGaghie,et al.  Simulation technology for health care professional skills training and assessment. , 1999, JAMA.

[63]  R A Miller,et al.  Effect of a Computer-Assisted General Medicine Diagnostic Consultation Service on Housestaff Diagnostic Strategy , 1989, Methods of Information in Medicine.

[64]  D. Bates,et al.  Prioritizing strategies for preventing medication errors and adverse drug events in pediatric inpatients. , 2003, Pediatrics.

[65]  J B Lemaire,et al.  Effectiveness of the Quick Medical Reference as a diagnostic tool. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[66]  P. Croskerry.  The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them , 2003, Academic medicine : journal of the Association of American Medical Colleges.

[67]  T. Brennan,et al.  The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. , 1991, The New England journal of medicine.

[68]  R A Miller,et al.  The Demise of the “Greek Oracle” Model for Medical Diagnostic Systems , 1990, Methods of Information in Medicine.

[69]  P Ramnarayan,et al.  ISABEL: a web-based differential diagnostic aid for paediatrics: results from an initial performance evaluation , 2003, Archives of disease in childhood.

[70]  Pat Croskerry,et al.  Cognitive forcing strategies in clinical decisionmaking. , 2003, Annals of emergency medicine.

[71]  D. Heckerman,et al.  Probabilistic diagnosis using a reformulation of the INTERNIST-1/QMR knowledge base. II. Evaluation of diagnostic performance. , 1991, Methods of information in medicine.

[72]  J P Kassirer,et al.  A report card on computer-assisted diagnosis--the grade: C. , 1994, The New England journal of medicine.