Use of Machine Learning to Identify Follow-Up Recommendations in Radiology Reports

Purpose: Assess follow-up recommendations in radiology reports, develop and assess traditional machine learning (TML) and deep learning (DL) models in identifying follow-up, and benchmark them against a natural language processing (NLP) system. Methods: This HIPAA-compliant, IRB approved study, was performed at an academic medical center generating >500,000 radiology reports annually. 1,000 randomly-selected ultrasound, x-ray, computed tomography and magnetic resonance imaging reports generated in 2016 were manually reviewed and annotated for follow-up recommendations. Traditional machine learning (Support Vector Machines, Random Forest, Logistic Regression) and deep learning (Recurrent Neural Nets) algorithms were constructed and trained on 850 reports (training data), with subsequent optimization of model architectures and parameters. Precision, recall and F1-score were calculated on the remaining 150 reports (test data). A previously-developed and validated NLP system (iSCOUT) was also applied to the test data, with equivalent metrics calculated. Results: 12.7% of reports had follow-up recommendations. The TML algorithms achieved F1 scores of 0.75 (Random Forest), 0.83 (Logistic Regression), and 0.85 (Support Vector Machine) on the test data. DL Recurrent Neural Nets had an F1 score of 0.71; iSCOUT also had an F1 score of 0.71. Performance of both TML and DL methods by F1-scores appeared to plateau after 500–700 samples while training. Conclusion: TML and DL are feasible methods to identify follow-up recommendations. These methods have great potential for near real-time monitoring of follow up recommendations in radiology reports.