A Conceptual Framework for Evaluating Information Technologies and Decision Support Systems for Bioterrorism Preparedness and Response

Objectives . The authors sought to develop a conceptual framework for evaluating whether existing information technologies and decision support systems (IT/DSSs) would assist the key decisions faced by clinicians and public health officials preparing for and responding to bioterrorism. Methods. They reviewed reports of natural and bioterrorism related infectious outbreaks, bioterrorism preparedness exercises, and advice from experts to identify the key decisions, tasks, and information needs of clinicians and public health officials during a bioterrorism response. The authors used task decomposition to identify the subtasks and data requirements of IT/DSSs designed to facilitate a bioterrorism response. They used the results of the task decomposition to develop evaluation criteria for IT/DSSs for bioterrorism preparedness. They then applied these evaluation criteria to 341 reports of 217 existing IT/DSSs that could be used to support a bioterrorism response. Main Results . In response to bioterrorism, clinicians must make decisions in 4 critical domains (diagnosis, management, prevention, and reporting to public health), and public health officials must make decisions in 4 other domains (interpretation of bioterrorism surveillance data, outbreak investigation, outbreak control, and communication). The time horizons and utility functions for these decisions differ. From the task decomposition, the authors identified critical subtasks for each of the 8 decisions. For example, interpretation of diagnostic tests is an important subtask of diagnostic decision making that requires an understanding of the tests’ sensitivity and specificity. Therefore, an evaluation criterion applied to reports of diagnostic IT/DSSs for bioterrorism asked whether the reports described the systems’ sensitivity and specificity. Of the 217 existing IT/DSSs that could be used to respond to bioterrorism, 79 studies evaluated 58 systems for at least 1 performance metric. Conclusions. The authors identified 8 key decisions that clinicians and public health officials must make in response to bioterrorism. When applying the evaluation system to 217 currently available IT/DSSs that could potentially support the decisions of clinicians and public health officials, the authors found that the literature provides little information about the accuracy of these systems.

[1]  G F Cooper,et al.  Representing and developing temporally abstracted knowledge as a means towards facilitating time modeling in medical decision-support systems , 1997, Comput. Biol. Medicine.

[2]  Evaluation of the Computer Program GIDEON for the Diagnosis of Fever in Patients Admitted to a Medical Service , 1998 .

[3]  Amy E. Smithson,et al.  Ataxia: The Chemical and Biological Terrorism Threat and the U , 1999 .

[4]  I Arene,et al.  Evaluation of quick medical reference (QMR) as a teaching tool. , 1998, M.D. computing : computers in medical practice.

[5]  F E Masarie,et al.  Quick medical reference (QMR) for diagnostic assistance. , 1986, M.D.Computing.

[6]  Philip K. Russell,et al.  Tularemia as a biological weapon: medical and public health management. , 2001, JAMA.

[7]  Peter J. Haug,et al.  Using medical language processing to support real-time evaluation of pneumonia guidelines , 2000, AMIA.

[8]  A. Donabedian,et al.  The quality of care. How can it be assessed? , 1988, JAMA.

[9]  R M Gardner,et al.  Computer surveillance of hospital-acquired infections and antibiotic use. , 1986, JAMA.

[10]  R M Gardner,et al.  Therapeutic antibiotic monitoring: surveillance using a computerized expert system. , 1990, The American journal of medicine.

[11]  W. Kapoor,et al.  A computer-assisted medical diagnostic consultation service. Implementation and prospective evaluation of a prototype. , 1989, Annals of internal medicine.

[12]  J. Myers,et al.  The INTERNIST-1/QUICK MEDICAL REFERENCE project--status report. , 1986, The Western journal of medicine.

[13]  A. L. Baker,et al.  Performance of four computer-based diagnostic systems. , 1994, The New England journal of medicine.

[14]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[15]  K Callan.  Preparing for a decision support system. , 2000, Topics in health information management.

[16]  Michael Gruninger,et al.  Methodology for the Design and Evaluation of Ontologies , 1995, IJCAI 1995.

[17]  Randolph A. Miller,et al.  User Variability in Abstracting and Entering Printed Case Histories with QUICK MEDICAL REFERENCE (QMR). , 1987 .

[18]  Andrew Friede,et al.  Application of Information Technology: CDC WONDER: A Cooperative Processing Architecture for Public Health , 1994, J. Am. Medical Informatics Assoc..

[19]  P. Lavori,et al.  Electronic Trial Banks: A Complementary Method for Reporting Randomized Trials , 2000, Medical decision making : an international journal of the Society for Medical Decision Making.

[20]  Update: Investigation of anthrax associated with intentional exposure and interim public health guidelines, October 2001. , 2001, MMWR. Morbidity and mortality weekly report.

[21]  Owens Dk,et al.  Computer-based decision support: wishing on a star? , 2001 .

[22]  Peter J. Haug,et al.  An integrated decision support system for diagnosing and managing patients with community-acquired pneumonia , 1999, AMIA.

[23]  Kim-Leng Poh,et al.  Time-critical dynamic decision modeling in medicine , 2002, Comput. Biol. Medicine.

[24]  Eta S. Berner,et al.  Effects of a decision support system on physicians' diagnostic performance. , 1999, Journal of the American Medical Informatics Association : JAMIA.

[25]  T. Clemmer,et al.  A computer-assisted management program for antibiotics and other antiinfective agents. , 1998, The New England journal of medicine.

[26]  D. S. Shapiro,et al.  Evaluation of the computer program GIDEON (Global Infectious Disease and Epidemiology Network) for the diagnosis of fever in patients admitted to a medical service. , 1998, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[27]  Gordon B. Hazen.  Stochastic Trees and the StoTree Modeling Environment: Models and Software for Medical Decision Analysis , 2004, Journal of Medical Systems.

[28]  R. Haynes,et al.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. , 1998, JAMA.

[29]  D. Owens,et al.  A Normative Analytic Framework for Development of Practice Guidelines for Specific Clinical Populations , 1997, Medical decision making : an international journal of the Society for Medical Decision Making.

[30]  D E Peterson,et al.  Massive outbreak of waterborne cryptosporidium infection in Milwaukee, Wisconsin: recurrence of illness and risk of secondary transmission. , 1995, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[31]  C. Becker.  20/20 hindsight. Months after anthrax claimed the lives of several. Americans, hospitals review their reaction to the event--and plan for future crises. , 2002, Modern healthcare.

[32]  R A Bankowitz,et al.  A prospective analysis of inter-rater agreement between a physician and a physician's assistant in selecting QMR vocabulary terms. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[33]  D K Owens,et al.  Use of Influence Diagrams to Structure Medical Decisions , 1997, Medical decision making : an international journal of the Society for Medical Decision Making.

[34]  C. P. Quinn,et al.  Bioterrorism-related inhalational anthrax: the first 10 cases reported in the United States. , 2001, Emerging infectious diseases.

[35]  G B Hazen,et al.  Factored Stochastic Trees , 1993, Medical decision making : an international journal of the Society for Medical Decision Making.

[36]  Todd R. Johnson,et al.  Generic tasks and task structures: history, critique and new directions , 1993 .

[37]  D. Josefson.  Toll of anthrax cases reaches 15 , 2001, BMJ : British Medical Journal.

[38]  Philip K. Russell,et al.  Smallpox as a biological weapon: medical and public health management. Working Group on Civilian Biodefense. , 1999, JAMA.

[39]  S. MacDonald.  What’s wrong with evaluating syndromic surveillance? , 2003, Journal of Urban Health.

[40]  Bonnie Kaplan,et al.  Evaluating informatics applications - clinical decision support systems literature review , 2001, Int. J. Medical Informatics.

[41]  A. Friede,et al.  CDC WONDER: a comprehensive on-line public health information system of the Centers for Disease Control and Prevention. , 1993, American journal of public health.

[42]  T. O'Toole,et al.  A plague on your city: observations from TOPOFF. , 2001, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[43]  M G Hunink,et al.  In Search of Tools to Aid Logical Thinking and Communicating about Medical Decision Making , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[44]  R S Evans,et al.  Improving empiric antibiotic selection using computer decision support. , 1994, Archives of internal medicine.

[45]  Philip K. Russell,et al.  Botulinum toxin as a biological weapon: medical and public health management. , 2001, JAMA.

[46]  R A Miller,et al.  Effect of a Computer-Assisted General Medicine Diagnostic Consultation Service on Housestaff Diagnostic Strategy , 1989, Methods of Information in Medicine.

[47]  D. Sosin.  Draft framework for evaluating syndromic surveillance systems , 2003, Journal of Urban Health.

[48]  Philip K. Russell,et al.  Plague as a biological weapon: medical and public health management. Working Group on Civilian Biodefense. , 2000, JAMA.

[49]  M. Musen,et al.  Bioterrorism preparedness and response: use of information technologies and decision support systems. , 2002, Evidence report/technology assessment.

[50]  D K Owens,et al.  Development of outcome-based practice guidelines: a method for structuring problems and synthesizing evidence. , 1993, The Joint Commission journal on quality improvement.

[51]  R. G. Parrish,et al.  Guidelines for evaluating surveillance systems. , 1988, MMWR supplements.

[52]  Computer-Based Decision Support : Wishing on a Star ? , .

[53]  David Classen,et al.  Implementing Antibiotic Practice Guidelines through Computer-Assisted Decision Support: Clinical and Financial Outcomes , 1996, Annals of Internal Medicine.

[54]  Allan S. Detsky,et al.  A randomized crossover trial of quick medical reference (QMR) as a teaching tool for medical interns , 1994, Journal of general internal medicine.

[55]  Ross D. Shachter,et al.  Representation and Analysis of Medical Decision Problems with Influence Diagrams , 1997, Medical decision making : an international journal of the Society for Medical Decision Making.

[56]  J. Bartlett,et al.  Death due to bioterrorism-related inhalational anthrax: report of 2 patients. , 2001, JAMA.

[57]  John Hornberger,et al.  Clinical Decision-support Systems: , 2000, Medical decision making : an international journal of the Society for Medical Decision Making.

[58]  G. Pugliese,et al.  Anthrax as a Biological Weapon: Medical and Public Health Management , 1999, Infection Control & Hospital Epidemiology.

[59]  E S Berner,et al.  Influence of case and physician characteristics on perceptions of decision support systems. , 1999, Journal of the American Medical Informatics Association : JAMIA.

[60]  C. Aydin,et al.  Evaluating Health Care Information Systems: Methods and Applications , 1993 .

[61]  R. Haynes,et al.  Effects of Computer-Based Clinical Decision Support Systems on Physician Performance and Patient Outcomes , 1998 .

[62]  R. Haynes,et al.  Effects of Computer-based Clinical Decision Support Systems on Clinician Performance and Patient Outcome: A Critical Appraisal of Research , 1994, Annals of Internal Medicine.

[63]  D. Pauze,et al.  Clinical presentation of inhalational anthrax following bioterrorism exposure: report of 2 surviving patients. , 2001, JAMA.

[64]  E. Shortliffe.  Computer programs to support clinical decision making. , 1990, JAMA.

[65]  A Donabedian,et al.  Quality Assessment and Monitoring , 1983, Evaluation & the health professions.

[66]  J B Lemaire,et al.  Effectiveness of the Quick Medical Reference as a diagnostic tool. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[67]  Elske Ammenwerth,et al.  Evaluation of health information systems - problems and challenges , 2003, Int. J. Medical Informatics.

[68]  Charles P. Friedman,et al.  Evaluation and Technology Assessment , 2001 .

[69]  R. E. Hoffman,et al.  Lessons learned from a full-scale bioterrorism exercise. , 2000, Emerging infectious diseases.

[70]  Silvana Quaglini,et al.  Deciding when to intervene: a Markov decision process approach , 2000, Int. J. Medical Informatics.

[71]  K. Adlassnig,et al.  Performance evaluation of medical expert systems using ROC curves. , 1989, Computers and biomedical research, an international journal.