MRI and CT bladder segmentation from classical to deep learning based approaches: Current limitations and lessons

Precise determination and assessment of bladder cancer (BC) extent of muscle invasion involvement guides proper risk stratification and personalized therapy selection. In this context, segmentation of both bladder walls and cancer are of pivotal importance, as it provides invaluable information to stage the primary tumor. Hence, multiregion segmentation on patients presenting with symptoms of bladder tumors using deep learning heralds a new level of staging accuracy and prediction of the biologic behavior of the tumor. Nevertheless, despite the success of these models in other medical problems, progress in multiregion bladder segmentation, particularly in MRI and CT modalities, is still at a nascent stage, with just a handful of works tackling a multiregion scenario. Furthermore, most existing approaches systematically follow prior literature in other clinical problems, without casting a doubt on the validity of these methods on bladder segmentation, which may present different challenges. Inspired by this, we provide an in-depth look at bladder cancer segmentation using deep learning models. The critical determinants for accurate differentiation of muscle invasive disease, current status of deep learning based bladder segmentation, lessons and limitations of prior work are highlighted.

[1]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[2]  C. Catalano,et al.  Overview of VI-RADS in Bladder Cancer. , 2020, AJR. American journal of roentgenology.

[3]  Eric Granger,et al.  Curriculum semi-supervised segmentation , 2019, MICCAI.

[4]  Cheng Chen,et al.  PnP-AdaNet: Plug-and-Play Adversarial Domain Adaptation Network at Unpaired Cross-Modality Cardiac Segmentation , 2019, IEEE Access.

[5]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[6]  Jose Dolz,et al.  Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision , 2020, MIDL.

[7]  Zhengrong Liang,et al.  A new partial volume segmentation approach to extract bladder wall for computer-aided detection in virtual cystoscopy , 2004, SPIE Medical Imaging.

[8]  Yaozong Gao,et al.  Fully convolutional networks for multi-modality isointense infant brain image segmentation , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[9]  Guoping Qiu,et al.  Bladder Cancer Multi-Class Segmentation in MRI With Pyramid-In-Pyramid Network , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[10]  Jose Dolz,et al.  IVD-Net: Intervertebral disc localization and segmentation in MRI with a multi-modal UNet , 2018, CSI@MICCAI.

[11]  Xin Yang,et al.  Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved? , 2018, IEEE Transactions on Medical Imaging.

[12]  Eric Granger,et al.  Constrained‐CNN losses for weakly supervised segmentation☆ , 2018, Medical Image Anal..

[13]  C. Catalano,et al.  Prospective Assessment of Vesical Imaging Reporting and Data System (VI-RADS) and Its Clinical Impact on the Management of High-risk Non-muscle-invasive Bladder Cancer Patients Candidate for Repeated Transurethral Resection. , 2019, European urology.

[14]  Giovanni Montana,et al.  Deep neural networks for anatomical brain segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[15]  Konstantinos Kamnitsas,et al.  Efficient multi‐scale 3D CNN with fully connected CRF for accurate brain lesion segmentation , 2016, Medical Image Anal..

[16]  M. Milowsky,et al.  Guideline on Muscle-Invasive and Metastatic Bladder Cancer (European Association of Urology Guideline): American Society of Clinical Oncology Clinical Practice Guideline Endorsement. , 2016, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[17]  Shuiwang Ji,et al.  Deep convolutional neural networks for multi-modality isointense infant brain image segmentation , 2015, NeuroImage.

[18]  Christian Igel,et al.  Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network , 2013, MICCAI.

[19]  H. C. Jones,et al.  The treatment of tumours of the bladder by transurethral resection. , 1962, British journal of urology.

[20]  Lubomir M. Hadjiiski,et al.  Segmentation of inner and outer bladder wall using deep-learning convolutional neural network in CT urography , 2017, Medical Imaging.

[21]  Xuelong Li,et al.  Adaptive Shape Prior Constrained Level Sets for Bladder MR Image Segmentation , 2014, IEEE Journal of Biomedical and Health Informatics.

[22]  J. Gschwend,et al.  An updated critical analysis of the treatment strategy for newly diagnosed high-grade T1 (previously T1G3) bladder cancer. , 2010, European urology.

[23]  Jose Dolz,et al.  Multi-Scale Self-Guided Attention for Medical Image Segmentation , 2021, IEEE Journal of Biomedical and Health Informatics.

[24]  Jose Dolz,et al.  3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study , 2016, NeuroImage.

[25]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[26]  Christophe De Vleeschouwer,et al.  Contour Propagation in CT Scans with Convolutional Neural Networks , 2018, ACIVS.

[27]  Yair Lotan,et al.  Treatment of Non‐Metastatic Muscle‐Invasive Bladder Cancer: AUA/ASCO/ASTRO/SUO Guideline , 2017, The Journal of urology.

[28]  Arjan Bel,et al.  Automatic bladder segmentation on CBCT for multiple plan ART of bladder cancer using a patient-specific bladder model , 2012, Physics in medicine and biology.

[29]  L. Schwartz,et al.  New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). , 2009, European journal of cancer.

[30]  M. Peyromaure,et al.  The value of a second transurethral resection in evaluating patients with bladder tumours. , 2003, European urology.

[31]  Antonio Pepe,et al.  PET-Train: Automatic Ground Truth Generation from PET Acquisitions for Urinary Bladder Segmentation in CT Images using Deep Learning , 2018, 2018 11th Biomedical Engineering International Conference (BMEiCON).

[32]  Deniz Erdogmus,et al.  Tversky Loss Function for Image Segmentation Using 3D Fully Convolutional Deep Networks , 2017, MLMI@MICCAI.

[33]  Jian Sun,et al.  ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Septimiu E. Salcudean,et al.  Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks , 2019, IEEE Transactions on Medical Imaging.

[35]  Xi Zhang,et al.  3D detection and extraction of bladder tumors via MR virtual cystoscopy , 2015, International Journal of Computer Assisted Radiology and Surgery.

[36]  J E Husband,et al.  Evaluation of the response to treatment of solid tumours – a consensus statement of the International Cancer Imaging Society , 2004, British Journal of Cancer.

[37]  Jizong Peng,et al.  Discretely-constrained deep network for weakly supervised segmentation , 2019, Neural Networks.

[38]  F. Khalifa,et al.  A Deep Learning-Based Approach for Accurate Segmentation of Bladder Wall using MR Images , 2019, 2019 IEEE International Conference on Imaging Systems and Techniques (IST).

[39]  D. Bluemke,et al.  Dynamic MRI of bladder cancer: evaluation of staging accuracy. , 2005, AJR. American journal of roentgenology.

[40]  Ke Wu,et al.  Bladder segmentation in MRI images using active region growing model , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[41]  K. Harada,et al.  Diagnostic Accuracy and Interobserver Agreement for the Vesical Imaging-Reporting and Data System for Muscle-invasive Bladder Cancer: A Multireader Validation Study. , 2019, European urology.

[42]  M. Viergever,et al.  Automatic Segmentation of MR Brain Images With a Convolutional Neural Network. , 2016, IEEE transactions on medical imaging.

[43]  Lubomir M. Hadjiiski,et al.  U‐Net based deep learning bladder segmentation in CT urography , 2019, Medical physics.

[44]  Lubomir M. Hadjiiski,et al.  Deep‐learning convolutional neural network: Inner and outer bladder wall segmentation in CT urography , 2019, Medical physics.

[45]  Zhengrong Liang,et al.  A unified EM approach to bladder wall segmentation with coupled level-set constraints , 2013, Medical Image Anal..

[46]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[47]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  K. Hammouda,et al.  A 3D CNN with a Learnable Adaptive Shape Prior for Accurate Segmentation of Bladder Wall Using MR Images , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[49]  Tanveer F. Syeda-Mahmood,et al.  3D Segmentation with Exponential Logarithmic Loss for Highly Unbalanced Object Sizes , 2018, MICCAI.

[50]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Lubomir M. Hadjiiski,et al.  Bladder Cancer Segmentation in CT for Treatment Response Assessment: Application of Deep-Learning Convolution Neural Network—A Pilot Study , 2016, Tomography.

[52]  Dimos Baltas,et al.  Esophagus segmentation in CT via 3D fully convolutional neural network and random walk , 2017, Medical physics.

[53]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[54]  Gang Li,et al.  Benchmark on Automatic Six-Month-Old Infant Brain Segmentation Algorithms: The iSeg-2017 Challenge , 2019, IEEE Transactions on Medical Imaging.

[55]  Jin Wang,et al.  The Diagnostic Value of MR Imaging in Differentiating T Staging of Bladder Cancer: A Meta-Analysis. , 2017, Radiology.

[56]  Yair Lotan,et al.  Nomogram for predicting disease recurrence after radical cystectomy for transitional cell carcinoma of the bladder. , 2006, The Journal of urology.

[57]  Fugen Zhou,et al.  Automatic bladder segmentation from CT images using deep CNN and 3D fully connected CRF-RNN , 2018, International Journal of Computer Assisted Radiology and Surgery.

[58]  J Alfred Witjes,et al.  Predicting recurrence and progression in individual patients with stage Ta T1 bladder cancer using EORTC risk tables: a combined analysis of 2596 patients from seven EORTC trials. , 2006, European urology.

[59]  Adel Said Elmaghraby,et al.  A CNN-Based Framework for Bladder Wall Segmentation Using MRI , 2019, 2019 Fifth International Conference on Advances in Biomedical Engineering (ICABME).

[60]  Zhengrong Liang,et al.  An Adaptive Window-Setting Scheme for Segmentation of Bladder Tumor Surface via MR Cystography , 2012, IEEE Transactions on Information Technology in Biomedicine.

[61]  K. Nikolaou,et al.  Imaging response assessment of immunotherapy in patients with renal cell and urothelial carcinoma , 2017, Current opinion in urology.

[62]  Iasonas Kokkinos,et al.  Sub-cortical brain structure segmentation using F-CNN'S , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[63]  João Manuel R. S. Tavares,et al.  Novel Approach to Segment the Inner and Outer Boundaries of the Bladder Wall in T2-Weighted Magnetic Resonance Images , 2011, Annals of Biomedical Engineering.

[64]  Jose Dolz,et al.  Constrained domain adaptation for segmentation , 2019, MICCAI.

[65]  V. Panebianco,et al.  Staging of bladder cancer with multiparametric MRI. , 2020, The British journal of radiology.

[66]  E. Sala,et al.  MRI of Bladder Cancer: Local and Nodal Staging. , 2020, Journal of magnetic resonance imaging : JMRI.

[67]  J. Cho,et al.  Diagnostic performance of MRI for prediction of muscle-invasiveness of bladder cancer: A systematic review and meta-analysis. , 2017, European journal of radiology.

[68]  G. Guazzoni,et al.  Role of Restaging Transurethral Resection for T1 Non-muscle invasive Bladder Cancer: A Systematic Review and Meta-analysis. , 2017, European urology focus.

[69]  J. Sethian,et al.  Fronts propagating with curvature-dependent speed: algorithms based on Hamilton-Jacobi formulations , 1988 .

[70]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[71]  Zhengrong Liang,et al.  Segmentation of multispectral bladder MR images with inhomogeneity correction for virtual cystoscopy , 2008, SPIE Medical Imaging.

[72]  Ismail Ben Ayed,et al.  On Regularized Losses for Weakly-supervised CNN Segmentation , 2018, ECCV.

[73]  Jing Yuan,et al.  Simultaneous Segmentation of Multiple Regions in 3D Bladder MRI by Efficient Convex Optimization of Coupled Surfaces , 2017, ICIG.

[74]  Christophe De Vleeschouwer,et al.  Using planning CTs to enhance CNN-based bladder segmentation on cone beam CT , 2019, Medical Imaging.

[75]  R. Huddart,et al.  Multiparametric Magnetic Resonance Imaging for Bladder Cancer: Development of VI-RADS (Vesical Imaging-Reporting And Data System). , 2018, European urology.

[76]  Jing Yuan,et al.  Multiregion segmentation of bladder cancer structures in MRI with progressive dilated convolutional networks , 2018, Medical physics.

[77]  J. Tavares,et al.  A versatile method for bladder segmentation in computed tomography two-dimensional images under adverse conditions , 2017, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[78]  Jose Dolz,et al.  Source-Relaxed Domain Adaptation for Image Segmentation , 2020, MICCAI.

[79]  Sébastien Ourselin,et al.  Automatic Brain Tumor Segmentation Using Cascaded Anisotropic Convolutional Neural Networks , 2017, BrainLes@MICCAI.

[80]  H. Mostafid,et al.  Measuring and improving the quality of transurethral resection for bladder tumour (TURBT) , 2012, BJU international.

[81]  Axel Hoos,et al.  Guidelines for the Evaluation of Immune Therapy Activity in Solid Tumors: Immune-Related Response Criteria , 2009, Clinical Cancer Research.

[82]  W. Horninger,et al.  Perforation during TUR of bladder tumours influences the natural history of superficial bladder cancer , 2014, World Journal of Urology.

[83]  Jing Yuan,et al.  HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation , 2018, IEEE Transactions on Medical Imaging.

[84]  C. Catalano,et al.  Multiparametric MRI of the bladder: inter-observer agreement and accuracy with the Vesical Imaging-Reporting and Data System (VI-RADS) at a single reference center , 2019, European Radiology.

[85]  Lubomir M. Hadjiiski,et al.  2D and 3D bladder segmentation using U-Net-based deep-learning , 2019, Medical Imaging.

[86]  Konstantinos Kamnitsas,et al.  DeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks , 2016, IEEE Transactions on Medical Imaging.

[87]  Christopher Joseph Pal,et al.  Brain tumor segmentation with Deep Neural Networks , 2015, Medical Image Anal..

[88]  Lubomir M. Hadjiiski,et al.  Urinary bladder segmentation in CT urography using deep-learning convolutional neural network and level sets. , 2016, Medical physics.

[89]  Sébastien Ourselin,et al.  Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations , 2017, DLMIA/ML-CDS@MICCAI.

[90]  Ben Glocker,et al.  Deformable medical image registration: setting the state of the art with discrete methods. , 2011, Annual review of biomedical engineering.

[91]  Pablo Piantanida,et al.  On Direct Distribution Matching for Adapting Segmentation Networks , 2019, MIDL.

[92]  P. Bhosale,et al.  Immunotherapy and the role of imaging , 2018, Cancer.

[93]  Zhengrong Liang,et al.  A Coupled Level Set Framework for Bladder Wall Segmentation With Application to MR Cystography , 2010, IEEE Transactions on Medical Imaging.

[94]  Michael Brady,et al.  Segmentation of the bladder wall using coupled level set methods , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[95]  Bo Du,et al.  Shape prior constrained PSO model for bladder wall MRI segmentation , 2017, Neurocomputing.

[96]  Keith J. Burnham,et al.  Automatic segmentation of clinical structures for RTP: Evaluation of a morphological approach , 2001 .