Evaluating Natural Language Processing Applications Applied to Outbreak and Disease Surveillance

Much of the pre-existing electronic data that could be harnessed for early outbreak detection is in free-text format. Natural language processing (NLP) techniques may be useful to biosurveillance by classifying and extracting information described in freetext sources. In the Real-time Outbreak and Disease Surveillance laboratory we are developing and evaluating NLP techniques for surveillance of syndromic presentations and specific findings or diseases potentially caused by bioterroristic or naturally-occurring outbreaks. We have implemented a three-stage evaluation process to determine whether NLP techniques are useful for outbreak detection. First, we are evaluating the technical accuracy of the NLP techniques to answer the question “How well can we classify, extract, or encode relevant information from text?” Second, we are evaluating the diagnostic accuracy of the techniques to answer the question “How well can we diagnose patients of interest using the NLP techniques?” Third, we are evaluating the outcome efficacy of the techniques to answer the question “How well can we detect outbreaks with an NLP-based biosurveillance system?” We give examples from our research for all three levels of evaluation and conclude with suggestions for determining whether NLP is feasible for outbreak and disease surveillance. Introduction and Background The appearance of new infectious diseases (e.g., the outbreak of Severe Acute Respiratory Syndrome (SARS) in Asia and Toronto), the reemergence of old infectious diseases (e.g., tuberculosis outbreaks), and the deliberate introduction of infectious diseases through bioterrorism (e.g., October 2001 anthrax attacks) demonstrate the need for surveillance of infectious disease [1]. The United States has not been prepared to deal with biological attacks [2], and biodefense has quickly become a national priority [3]. In response to the need for better biodefense, several research groups have developed electronic surveillance systems [4-13] that monitor a variety of different sources of data including over-the-counter drug sales [14, 15], web-based physician entry of reports [16], 911 calls [17], consumer health hotline telephone calls [18, 19], and ambulatory care visit records [20-24]. Many of the systems monitor pre-existing electronic ED data [25] that typically include date of admission, sex, age, address, coded discharge diagnosis [22, 23, 26], and free-text triage chief complaint [21, 27-30]. Detection algorithms count the number of occurrences of a variable or a combination of variables in a given spatial location over a given time period to look for anomalous patterns [21, 31-33]. If the algorithms detect a significant increase in a given variable, such as the number of patients with a gastrointestinal illness, the detection algorithms alarm relevant medical and public health officials of a possible outbreak. The Real-time Outbreak and Disease Surveillance (RODS) system [34] is a biosurveillance system adherent to the CDC’s NEDSS standards [35] that was developed in 1999 at the University of Pittsburgh and is currently deployed in four states, including Pennsylvania, Utah, New Jersey, and Ohio. For over 100 hospitals in the four states, RODS collects real-time admission data, including age, sex, zip code, and triage chief complaint. Time-series detection algorithms are applied to the information in the database, and the counts of patients with seven types of syndromic presentations are shown in graphical form on the user interface, shown in Figure 1. The interface also includes a geographic information system that shows counts of syndromic presentations by zip code. If the actual number of patients presenting with gastrointestinal complaints, for instance, exceeds the number expected in a given geographical location over a given time period, RODS’ notification subsystem sends an electronic alarm to a team of researchers and public health physicians for possible investigation. RODS software is currently open source [36] and is available for free download at www.health.pitt.edu/rods/sw. Input variables for the detection algorithms in RODS and other biosurveillance systems must be coded data, i.e., data stored in a format that can be interpreted by a computer. For example, a biosurveillance system may monitor the number of patients with Pneumonia, which may indicate a possible outbreak of Influenza, SARS, or inhalational Anthrax. Researchers in medical informatics have developed electronic diagnostic systems integrating multiple sources of data from a patient’s medical record to generate a probability that a patient has Pneumonia [37-39]. The variables required to determine the probability of Pneumonia include age and risk factors, vital signs, symptoms and physical findings, laboratory results, blood gas levels, and chest radiograph results. Values for Figure 1. Interface for Real-time Outbreak and Disease (RODS) system, showing syndromic classifications over a one-week period for all admissions in the specificed jurisdiction. some of the variables are stored in hospital information systems in coded format; however, variables involving a patient’s symptoms and physical findings, such as cough or adventitious respiratory sounds, and the results of a chest radiograph are usually stored as dictated reports in uncoded, free-text format. To use these variables for computerized decision support, the variables must be encoded from the textual reports. A physician reading the reports could easily determine the correct values for the variables. However, paying physicians to encode reports is impractical. A more feasible solution is applying natural language processing (NLP) techniques to convert the freetext data into an encoded representation that can be used for later inference [40]. Over the last few decades the medical informatics community has actively applied NLP techniques to the medical domain [41, 42]. The Linguistic String Project developed one of the first medical NLP systems that included comprehensive semantic and syntactic knowledge [43-49] that has also been ported to French and German [50-55]. Columbia Presbyterian Medical Center has evaluated and deployed a system called MedLEE [56-60] that extracts clinical information from radiology reports, discharge summaries, visit notes, electrocardiography, echocardiography, and pathology notes. MedLEE has been shown to be as accurate as physicians at extracting clinical concepts from chest radiograph reports [61, 62] and has been evaluated for a variety of applications including detecting patients with suspected tuberculosis [63-65], identifying findings suspicious for breast cancer [66], stroke [67], and community acquired Pneumonia [68], and deriving co morbidities from text [69]. Other medical informatics research groups have also created and evaluated NLP systems for extracting clinical information from medical texts and have shown them to be accurate in limited domains [70-86]. NLP techniques have been used for a variety of applications including quality assessment in radiology [87, 88], identification of structures in radiology images [89, 90], facilitation of structured reporting [72, 91] and order entry [92, 93], and encoding variables required by automated decision support systems such as guidelines [94], diagnostic systems [95], and antibiotic therapy alarms [96]. NLP has only recently been applied to the domain of outbreak and disease surveillance, and most of the research has focused on processing free-text chief complaints recorded in the emergency department [97-102]. We have applied NLP techniques to chief complaints, ED reports, and chest radiograph reports in order to acquire coded variables that may be useful in outbreak detection. To quantify the value of NLP in the domain of biosurveillance, we have adapted a hierarchical model of technology assessment from the domain of medical imaging, described by Thornbury and Fryback [103]. First, we have evaluated the technical accuracy of our NLP techniques to answer the question “How well can we classify, extract, or encode relevant information from text?” Second, we have evaluated the diagnostic accuracy of the techniques to answer the question “How well can we diagnose patients of interest using the NLP techniques?” Third, we have evaluated the outcome efficacy of the techniques to answer the question “How well can we detect outbreaks with an NLP-based biosurveillance system?” In the Methods section we describe the three levels of evaluation, using the hypothetical example of Pneumonia surveillance as an example. We briefly describe studies we have performed to evaluate NLP technologies for all three levels of evaluation and provide references for details about the studies. In the Results section we provide results from our research for the three levels of evaluation. In the Discussion section, we discuss implications of our findings and suggest three points to consider when appraising the feasibility of applying NLP to the domain of outbreak detection.