Multi-task learning for segmentation and classification of tumors in 3D automated breast ultrasound images

Abstract Tumor classification and segmentation are two important tasks for computer-aided diagnosis (CAD) using 3D automated breast ultrasound (ABUS) images. However, they are challenging due to the significant shape variation of breast tumors and the fuzzy nature of ultrasound images (e.g., low contrast and signal to noise ratio). Considering the correlation between tumor classification and segmentation, we argue that learning these two tasks jointly is able to improve the outcomes of both tasks. In this paper, we propose a novel multi-task learning framework for joint segmentation and classification of tumors in ABUS images. The proposed framework consists of two sub-networks: an encoder-decoder network for segmentation and a light-weight multi-scale network for classification. To account for the fuzzy boundaries of tumors in ABUS images, our framework uses an iterative training strategy to refine feature maps with the help of probability maps obtained from previous iterations. Experimental results based on a clinical dataset of 170 3D ABUS volumes collected from 107 patients indicate that the proposed multi-task framework improves tumor segmentation and classification over the single-task learning counterparts.

[1]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Yazhu Chen,et al.  Measuring shape complexity of breast lesions on ultrasound images , 2008, SPIE Medical Imaging.

[4]  William Schroeder,et al.  The Visualization Toolkit: An Object-Oriented Approach to 3-D Graphics , 1997 .

[5]  Ruey-Feng Chang,et al.  Computer-aided diagnosis for the classification of breast masses in automated whole breast ultrasound images. , 2011, Ultrasound in medicine & biology.

[6]  Andriy I. Bandos,et al.  A Prospective Study of Automated Breast Ultrasound Screening of Women with Dense Breasts in a Digital Breast Tomosynthesis-based Practice. , 2020, Journal of breast imaging.

[7]  Jayanthi Sivaswamy,et al.  A Deep Learning based Joint Segmentation and Classification Framework for Glaucoma Assesment in Retinal Color Fundus Images , 2018, ArXiv.

[8]  Walid Al-Dhabyani,et al.  Deep Learning Approaches for Data Augmentation and Classification of Breast Masses using Ultrasound Images , 2019, International Journal of Advanced Computer Science and Applications.

[9]  Klaus J. Kirchberg,et al.  Robust Face Detection Using the Hausdorff Distance , 2001, AVBPA.

[10]  A. Jemal,et al.  Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries , 2018, CA: a cancer journal for clinicians.

[11]  Wilfrido Gómez-Flores,et al.  Improving classification performance of breast lesions on ultrasonography , 2015, Pattern Recognit..

[12]  Xiaolan Tang,et al.  Breast Ultrasound Image Classification and Segmentation Using Convolutional Neural Networks , 2018, PCM.

[13]  Doina Precup,et al.  Exploring uncertainty measures in deep networks for Multiple sclerosis lesion detection and segmentation , 2020, Medical Image Anal..

[14]  A Vourtsis,et al.  Three-dimensional automated breast ultrasound: Technical aspects and first results. , 2019, Diagnostic and interventional imaging.

[15]  Daniel Rueckert,et al.  Multi-Task Learning for Left Atrial Segmentation on GE-MRI , 2018, STACOM@MICCAI.

[16]  Ran Li,et al.  Self-co-attention neural network for anatomy segmentation in whole breast ultrasound , 2020, Medical Image Anal..

[17]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[18]  L Leija,et al.  Computerized lesion segmentation of breast ultrasound based on marker-controlled watershed transformation. , 2009, Medical physics.

[19]  Dimitris N. Metaxas,et al.  Joint Segmentation and Fine-Grained Classification of Nuclei in Histopathology Images , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[20]  M. Giger,et al.  Automated Breast Ultrasound in Breast Cancer Screening of Women With Dense Breasts: Reader Study of Mammography-Negative and Mammography-Positive Cancers. , 2016, AJR. American journal of roentgenology.

[21]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[22]  Yi Wang,et al.  Breast Cancer Classification in Automated Breast Ultrasound Using Multiview Convolutional Neural Network with Transfer Learning. , 2020, Ultrasound in medicine & biology.

[23]  Purang Abolmaesumi,et al.  Ultrasound RF Time Series for Classification of Breast Lesions , 2015, IEEE Transactions on Medical Imaging.

[24]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Wei Shen,et al.  Multi-scale Convolutional Neural Networks for Lung Nodule Classification , 2015, IPMI.

[26]  Remco Duits,et al.  Automated Segmentation of Lesions in Ultrasound Using Semi-pixel-wise Cycle Generative Adversarial Nets , 2019, ArXiv.

[27]  Z. Jane Wang,et al.  An End-to-End Multi-Task Deep Learning Framework for Skin Lesion Analysis , 2020, IEEE Journal of Biomedical and Health Informatics.

[28]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Joanna Czajkowska,et al.  Breast Cancer Segmentation Method in Ultrasound Images , 2017 .

[30]  David Gur,et al.  Digital Mammography versus Digital Mammography Plus Tomosynthesis in Breast Cancer Screening: The Oslo Tomosynthesis Screening Trial. , 2019, Radiology.

[31]  Chia-Yen Lee,et al.  Fully automated lesion segmentation and visualization in automated whole breast ultrasound (ABUS) images. , 2020, Quantitative imaging in medicine and surgery.

[32]  Dinggang Shen,et al.  Densely Deep Supervised Networks with Threshold Loss for Cancer Detection in Automated Breast Ultrasound , 2018, MICCAI.

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Cicero Ferreira Fernandes Costa Filho,et al.  Breast Lesion Classification in Ultrasound Images Using Deep Convolutional Neural Network , 2020, IEEE Access.

[35]  Houjin Chen,et al.  Uncertainty Aware Temporal-Ensembling Model for Semi-Supervised ABUS Mass Segmentation , 2020, IEEE Transactions on Medical Imaging.

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Tao Tan,et al.  Classification of breast mass in 3D ultrasound images with annotations based on convolutional neural networks , 2018 .

[38]  Jakub Nalepa,et al.  Data Augmentation for Brain-Tumor Segmentation: A Review , 2019, Front. Comput. Neurosci..

[39]  Rachel F Brem,et al.  Screening breast ultrasound: past, present, and future. , 2015, AJR. American journal of roentgenology.

[40]  Vishal M. Patel,et al.  Simultaneous Segmentation and Classification of Bone Surfaces from Ultrasound Using a Multi-feature Guided CNN , 2018, MICCAI.

[41]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  J. Beck,et al.  Time to Surgery and Breast Cancer Survival in the United States. , 2016, JAMA oncology.

[43]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[44]  F J Gilbert,et al.  Artificial intelligence in breast imaging. , 2019, Clinical radiology.

[45]  M. Soryani,et al.  Mass Segmentation in Automated 3-D Breast Ultrasound Using Adaptive Region Growing and Supervised Edge-Based Deformable Model , 2018, IEEE Transactions on Medical Imaging.

[46]  Vivek Kumar Singh,et al.  An Efficient Solution for Breast Tumor Segmentation and Classification in Ultrasound Images Using Deep Adversarial Learning , 2019, ArXiv.

[47]  Jie-Zhi Cheng,et al.  Computer-aided US diagnosis of breast lesions by using cell-based contour grouping. , 2010, Radiology.

[48]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[49]  Hang Li,et al.  Segmentation of breast anatomy for automated whole breast ultrasound images with boundary regularized convolutional encoder-decoder network , 2018, Neurocomputing.

[50]  Fang-Cheng Yeh,et al.  ACCOMP: Augmented cell competition algorithm for breast lesion demarcation in sonography. , 2010, Medical physics.

[51]  Fang-Cheng Yeh,et al.  Cell-competition algorithm: a new segmentation algorithm for multiple objects with irregular boundaries in ultrasound images. , 2005, Ultrasound in medicine & biology.

[52]  Pheng-Ann Heng,et al.  Cascaded Fully Convolutional Networks for automatic prenatal ultrasound image segmentation , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[53]  Yi-Hong Chou,et al.  Boundary Regularized Convolutional Neural Network for Layer Parsing of Breast Anatomy in Automated Whole Breast Ultrasound , 2017, MICCAI.

[54]  Jeon-Hor Chen,et al.  Computer-aided classification of breast masses using speckle features of automated breast ultrasound images. , 2012, Medical physics.

[55]  Zhuowen Tu,et al.  Auto-Context and Its Application to High-Level Vision Tasks and 3D Brain Image Segmentation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  C. Zuiani,et al.  Automated breast ultrasound: basic principles and emerging clinical applications , 2017, La radiologia medica.

[57]  Wei Liu,et al.  ParseNet: Looking Wider to See Better , 2015, ArXiv.

[58]  Jian Shen,et al.  Medical image classification based on multi-scale non-negative sparse coding , 2017, Artif. Intell. Medicine.