Research Paper: Automatic Detection of Acute Bacterial Pneumonia from Chest X-ray Reports

OBJECTIVE To evaluate the performance of a natural language processing system in extracting pneumonia-related concepts from chest x-ray reports. METHODS DESIGN Four physicians, three lay persons, a natural language processing system, and two keyword searches (designated AAKS and KS) detected the presence or absence of three pneumonia-related concepts and inferred the presence or absence of acute bacterial pneumonia from 292 chest x-ray reports. Gold standard: Majority vote of three independent physicians. Reliability of the gold standard was measured. OUTCOME MEASURES Recall, precision, specificity, and agreement (using Finn's R: statistic) with respect to the gold standard. Differences between the physicians and the other subjects were tested using the McNemar test for each pneumonia concept and for the disease inference of acute bacterial pneumonia. RESULTS Reliability of the reference standard ranged from 0.86 to 0.96. Recall, precision, specificity, and agreement (Finn R:) for the inference on acute bacterial pneumonia were, respectively, 0.94, 0.87, 0.91, and 0.84 for physicians; 0.95, 0.78, 0.85, and 0.75 for natural language processing system; 0.46, 0.89, 0.95, and 0.54 for lay persons; 0.79, 0.63, 0.71, and 0.49 for AAKS; and 0.87, 0.70, 0.77, and 0.62 for KS. The McNemar pairwise comparisons showed differences between one physician and the natural language processing system for the infiltrate concept and between another physician and the natural language processing system for the inference on acute bacterial pneumonia. The comparisons also showed that most physicians were significantly different from the other subjects in all pneumonia concepts and the disease inference. CONCLUSION In extracting pneumonia related concepts from chest x-ray reports, the performance of the natural language processing system was similar to that of physicians and better than that of lay persons and keyword searches. The encoded pneumonia information has the potential to support several pneumonia-related applications used in our institution. The applications include a decision support system called the antibiotic assistant, a computerized clinical protocol for pneumonia, and a quality assurance application in the radiology department.

[1]  L M Lau,et al.  A natural language understanding system combining syntactic and semantic techniques. , 1994, Proceedings. Symposium on Computer Applications in Medical Care.

[2]  H Brenner,et al.  Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. , 1997, Statistics in medicine.

[3]  Peter J. Haug,et al.  Comparing expert systems for identifying chest x-ray reports that support pneumonia , 1999, AMIA.

[4]  Pryor Ta,et al.  The HELP medical record system. , 1988 .

[5]  Grover J. Whitehurst,et al.  Interrater agreement for journal manuscript reviews. , 1984 .

[6]  J C Hershey.  Sensitivity Analysis in Medical Decision Making , 1991, Medical decision making : an international journal of the Society for Medical Decision Making.

[7]  James F. Allen.  Natural language understanding , 1987, Bejnamin/Cummings series in computer science.

[8]  L A Lenert,et al.  Monitoring free-text data using medical language processing. , 1993, Computers and biomedical research, an international journal.

[9]  Guidelines for the initial Management of Adults with Community-acquired Pneumoma : Diagnosis , Assessment of Severity ; and initial Antimicrobial fieraky , 1998 .

[10]  S Hughes.  Bedside terminals: CliniCom. , 1988, M.D. computing : computers in medical practice.

[11]  Victor L. Yu,et al.  Medical Section pf the American Lung Association: Guidelines for the Initial Management of Adults with Community-acquired Pneumonia: Diagnosis, Assessment of Severity, and Initial Antimicrobial Therapy , 1993 .

[12]  P J Haug,et al.  Quality Control in a Medical Information System , 1991, Medical decision making : an international journal of the Society for Medical Decision Making.

[13]  R S Evans,et al.  Development of an automated antibiotic consultant. , 1993, M.D. computing : computers in medical practice.

[14]  P J Haug,et al.  Development and evaluation of a computerized admission diagnoses encoding system. , 1996, Computers and biomedical research, an international journal.

[15]  Peter J. Haug,et al.  Diagnosing community-acquired pneumonia with a Bayesian network , 1998, AMIA.

[16]  D Ozonoff,et al.  Disagreements in chest roentgen interpretation. , 1975, Chest.

[17]  Peter J. Haug,et al.  Automatic extraction of PIOPED interpretations from ventilation/perfusion lung scan reports , 1998, AMIA.

[18]  T. Clemmer,et al.  A computer-assisted management program for antibiotics and other antiinfective agents. , 1998, The New England journal of medicine.

[19]  George Hripcsak,et al.  Research Paper: A Reliability Study for Evaluating Information Extraction from Radiology Reports , 1999, J. Am. Medical Informatics Assoc..

[20]  L. Koran,et al.  The reliability of clinical methods, data and judgments (first of two parts). , 1975, The New England journal of medicine.

[21]  Peter J. Haug,et al.  Development and evaluation of a computerized admission diagnoses encoding system. , 1996, Computers and biomedical research, an international journal.

[22]  Naomi Sager,et al.  Research Paper: Natural Language Processing and the Representation of Clinical Data , 1994, J. Am. Medical Informatics Assoc..

[23]  Evlin L. Kinney,et al.  Clinical Epidemiology and Biostatistics: A Primer for Clinical Investigators and Decision-Makers , 1989 .

[24]  A. Hasman,et al.  Probabilistic reasoning in intelligent systems: Networks of plausible inference , 1991 .

[25]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[26]  V L Yu,et al.  Guidelines for the initial management of adults with community-acquired pneumonia: diagnosis, assessment of severity, and initial antimicrobial therapy. American Thoracic Society. Medical Section of the American Lung Association. , 1993 .

[27]  Carol Friedman,et al.  Research Paper: A General Natural-language Text Processor for Clinical Radiology , 1994, J. Am. Medical Informatics Assoc..

[28]  W. DuMouchel,et al.  Unlocking Clinical Data from Narrative Reports: A Study of Natural Language Processing , 1995, Annals of Internal Medicine.

[29]  George Hripcsak,et al.  Classification algorithms applied to narrative reports , 1999, AMIA.

[30]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[31]  M. Fine,et al.  Assessing prognosis and selecting an initial site of care for adults with community-acquired pneumonia. , 1998, Infectious disease clinics of North America.

[32]  P J Haug,et al.  Experience with a mixed semantic/syntactic parser. , 1995, Proceedings. Symposium on Computer Applications in Medical Care.