Using medical language processing to support real-time evaluation of pneumonia guidelines

OBJECTIVE To evaluate if a medical language processing (MLP) system is able to support real-time computerization of community-acquired pneumonia (CAP) guidelines. METHODS Prospective validation study in the emergency department of a tertiary care facility. All the chest x-ray reports available in real-time for an admission decision during a five-week period were included. The MLP system was compared to a physician for the automatic selection of eligible patients and on the extraction of radiographic findings required by five different CAP guidelines. The gold standard comprised of three independent physicians and reliability measures were calculated. The outcome measures were the area under the receiver operated characteristic curve (AUC) for selecting eligible patients, sensitivity, positive predictive value (PPV), and specificity for the extraction of radiographic findings. RESULTS During the five-week period, 243 reports were available in real-time. The AUCs on selecting eligible CAP patients were 89.7% (CI: 84.2%, 93.7%) for the MLP system, and 93.3% (CI: 83.9%, 97.8%) for the physician. The average sensitivity, PPV, and specificity for radiographic findings that assessed localization and extension of CAP were respectively: 94%, 87%, 96% (physician); and 34%, 90%, 95% (MLP system). Both, the MLP system and the physician had average sensitivity, PPV, and specificity of 97%, 97%, and 99%, respectively, when localization was not an issue. Reliability measures for the gold standard were above 70%. CONCLUSION The MLP system was able to support real-time computerization of guidelines by selecting eligible patients and extracting radiographic findings that do not assess localization and extension of CAP.