Automated annotation and classification of BI-RADS assessment from radiology reports

The Breast Imaging Reporting and Data System (BI-RADS) was developed to reduce variation in the descriptions of findings. Manual analysis of breast radiology report data is challenging but is necessary for clinical and healthcare quality assurance activities. The objective of this study is to develop a natural language processing (NLP) system for automated BI-RADS categories extraction from breast radiology reports. We evaluated an existing rule-based NLP algorithm, and then we developed and evaluated our own method using a supervised machine learning approach. We divided the BI-RADS category extraction task into two specific tasks: (1) annotation of all BI-RADS category values within a report, (2) classification of the laterality of each BI-RADS category value. We used one algorithm for task 1 and evaluated three algorithms for task 2. Across all evaluations and model training, we used a total of 2159 radiology reports from 18 hospitals, from 2003 to 2015. Performance with the existing rule-based algorithm was not satisfactory. Conditional random fields showed a high performance for task 1 with an F-1 measure of 0.95. Rules from partial decision trees (PART) algorithm showed the best performance across classes for task 2 with a weighted F-1 measure of 0.91 for BIRADS 0-6, and 0.93 for BIRADS 3-5. Classification performance by class showed that performance improved for all classes from Naïve Bayes to Support Vector Machine (SVM), and also from SVM to PART. Our system is able to annotate and classify all BI-RADS mentions present in a single radiology report and can serve as the foundation for future studies that will leverage automated BI-RADS annotation, to provide feedback to radiologists as part of a learning health system loop.

[1]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[2]  Hua Xu,et al.  A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries , 2011, J. Am. Medical Informatics Assoc..

[3]  William W. Stead,et al.  Toward a science of learning systems: a research agenda for the high-functioning Learning Health System , 2014, J. Am. Medical Informatics Assoc..

[4]  D. Berry,et al.  Effect of screening and adjuvant therapy on mortality from breast cancer , 2005 .

[5]  Dimitrios Mitsouras,et al.  Natural Language Processing Technologies in Radiology Research and Clinical Applications. , 2016, Radiographics : a review publication of the Radiological Society of North America, Inc.

[6]  Siddhartha Jonnalagadda,et al.  Coreference analysis in clinical notes: a multi-pass sieve with alternate anaphora resolution modules , 2012, J. Am. Medical Informatics Assoc..

[7]  A. Jemal,et al.  Global Cancer Statistics , 2011 .

[8]  G. Badan,et al.  Complete internal audit of a mammography service in a reference institution for breast imaging* , 2014, Radiologia brasileira.

[9]  Alyssa Cwanger,et al.  Bayesian Probability of Malignancy With BI‐RADS Sonographic Features , 2014, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[10]  M.K. Markey,et al.  Bayesian networks of BI-RADS/spl trade/ descriptors for breast lesion classification , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[11]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[12]  Bethany Percha Machine Learning Approaches to Automatic BI-RADS Classification of Mammography Reports , 2010 .

[13]  David Page,et al.  Extracting BI-RADS features from Portuguese clinical texts , 2012, 2012 IEEE International Conference on Bioinformatics and Biomedicine.

[14]  Halimat Jumai Akande,et al.  A five year audit of mammography in a tertiary hospital, North Central Nigeria , 2015, Nigerian medical journal : journal of the Nigeria Medical Association.

[15]  Timothy J Wilt,et al.  Screening for breast cancer: U.S. Preventive Services Task Force recommendation statement. , 2009, Annals of internal medicine.

[16]  J. Ferlay,et al.  Global Cancer Statistics, 2002 , 2005, CA: a cancer journal for clinicians.

[17]  Guergana K. Savova,et al.  Discerning Tumor Status from Unstructured MRI Reports—Completeness of Information in Existing Reports and Utility of Automated Natural Language Processing , 2009, Journal of Digital Imaging.

[19]  Min Li,et al.  High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge , 2010, J. Am. Medical Informatics Assoc..

[20]  Bethany Percha,et al.  Automatic classification of mammography reports by BI-RADS breast tissue composition class , 2012, J. Am. Medical Informatics Assoc..

[21]  Martha Palmer,et al.  Reducing the Need for Double Annotation , 2011, Linguistic Annotation Workshop.

[22]  Thomas Lavergne,et al.  Natural language processing of radiology reports for the detection of thromboembolic diseases and clinically relevant incidental findings , 2014, BMC Bioinformatics.

[23]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[24]  D. Vanel The American College of Radiology (ACR) Breast Imaging and Reporting Data System (BI-RADS): a step towards a universal radiological language? , 2007, European journal of radiology.

[25]  A. Jemal,et al.  Cancer statistics, 2016 , 2016, CA: a cancer journal for clinicians.

[26]  Ramin Khorasani,et al.  Automated Extraction of BI-RADS Final Assessment Categories from Radiology Reports with Natural Language Processing , 2013, Journal of Digital Imaging.

[27]  Selen Bozkurt,et al.  Using automatically extracted information from mammography reports for decision-support , 2016, J. Biomed. Informatics.

[28]  Hongfang Liu,et al.  Using machine learning for concept extraction on clinical documents from multiple data sources , 2011, J. Am. Medical Informatics Assoc..

[29]  E. Burnside,et al.  Development of an online, publicly accessible naive Bayesian decision support tool for mammographic mass lesions based on the American College of Radiology (ACR) BI-RADS lexicon , 2015, European Radiology.

[30]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[31]  S. Sathiya Keerthi,et al.  Improvements to Platt's SMO Algorithm for SVM Classifier Design , 2001, Neural Computation.

[32]  S. Orel,et al.  BI-RADS categorization as a predictor of malignancy. , 1999, Radiology.

[33]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[34]  Karla Kerlikowske,et al.  Mammography Surveillance Following Breast Cancer , 2003, Breast Cancer Research and Treatment.

[35]  Nuno A. Fonseca,et al.  Predicting Malignancy from Mammography Findings and Surgical Biopsies , 2011, 2011 IEEE International Conference on Bioinformatics and Biomedicine.

[36]  M. Lacquement,et al.  Positive predictive value of the Breast Imaging Reporting and Data System. , 1999, Journal of the American College of Surgeons.

[37]  A. Tosteson,et al.  Mammography in 53,803 women from the New Hampshire mammography network. , 2000, Radiology.

[38]  Sunghwan Sohn,et al.  Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications , 2010, J. Am. Medical Informatics Assoc..

[39]  D. Hovsepian,et al.  The joint commission practice performance evaluation: a primer for radiologists. , 2010, Journal of the American College of Radiology : JACR.

[40]  Timothy J Wilt,et al.  Screening for breast cancer: U.S. Preventive Services Task Force recommendation statement. , 2009, Annals of internal medicine.

[41]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[42]  Ian H. Witten,et al.  Generating Accurate Rule Sets Without Global Optimization , 1998, ICML.

[43]  D. Berry,et al.  Effect of screening and adjuvant therapy on mortality from breast cancer. , 2006, The New England journal of medicine.

[44]  Michael Feldman,et al.  caTIES: a grid based system for coding and retrieval of surgical pathology reports and tissue specimens in support of translational research , 2010, J. Am. Medical Informatics Assoc..

[45]  Loes M. M. Braun,et al.  Natural Language Processing in Radiology: A Systematic Review. , 2016, Radiology.

[46]  Rebecca L. Siegel Mph,et al.  Cancer statistics, 2016 , 2016 .

[47]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[48]  A. Jemal,et al.  Global cancer statistics, 2012 , 2015, CA: a cancer journal for clinicians.

[49]  Özlem Uzuner,et al.  Extracting medication information from clinical text , 2010, J. Am. Medical Informatics Assoc..

[50]  J. Ramon,et al.  Machine learning techniques to examine large patient databases. , 2009, Best practice & research. Clinical anaesthesiology.

[51]  McGinnis Jm,et al.  The learning healthcare system : workshop summary , 2007 .

[52]  Hong Yu,et al.  Learning for Biomedical Information Extraction: Methodological Review of Recent Advances , 2016, ArXiv.

[53]  David Page,et al.  Information Extraction for Clinical Data Mining: A Mammography Case Study , 2009, 2009 IEEE International Conference on Data Mining Workshops.

[54]  Anthony N. Nguyen,et al.  Automatic Classification of Free-Text Radiology Reports to Identify Limb Fractures using Machine Learning and the SNOMED CT Ontology , 2013, AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science.

[55]  J. Lortet-Tieulent,et al.  Breast Cancer Screening for Women at Average Risk: 2015 Guideline Update From the American Cancer Society. , 2015, JAMA.

[56]  Jun'ichi Tsujii,et al.  Named entity recognition of follow-up and time information in 20 000 radiology reports , 2012, J. Am. Medical Informatics Assoc..

[57]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[58]  Carol Friedman,et al.  Identification of findings suspicious for breast cancer based on natural language processing of mammogram reports , 1997, AMIA.

[59]  Will Styler,et al.  Anafora: A Web-based General Purpose Annotation Tool , 2013, HLT-NAACL.

[60]  Selen Bozkurt,et al.  Automatic abstraction of imaging observations with their characteristics from mammography reports , 2015, J. Am. Medical Informatics Assoc..

[61]  M. Eberl,et al.  BI-RADS classification for management of abnormal mammograms. , 2006, Journal of the American Board of Family Medicine : JABFM.

[62]  Selen Bozkurt,et al.  Automated detection of ambiguity in BI-RADS assessment categories in mammography reports. , 2014, Studies in health technology and informatics.

[63]  Joseph Y. Lo,et al.  Bayesian networks of BI-RADS™ descriptors for breast lesion Classification , 2004 .

[64]  Hongyuan Gao,et al.  Using natural language processing to extract mammographic findings , 2015, J. Biomed. Informatics.