Multi-needle detection in ultrasound image using max-margin mask R-CNN

Digitalizing all the needles in ultrasound (US) images is a crucial step of treatment planning for US-guided high-dose-rate (HDR) prostate brachytherapy. However, current computer-aided technologies are broadly focused on single-needle digitization, while manual digitization of all needles is labor intensive and time consuming. In this paper, we proposed a deep learning-based workflow for fast automatic multi-needle digitization, including needle shaft detection and needle tip detection. The major workflow is composed of two components: a large margin mask R-CNN model (LMMask R-CNN), which adopts the lager margin loss to reformulate Mask R-CNN for needle shaft localization, and a needle-based density-based spatial clustering of application with noise (DBSCAN) algorithm which integrates priors to model a needle in an iteration for a needle shaft refinement and tip detections. Besides, we use the skipping connection in neural network architecture to improve the supervision in hidden layers. Our workflow was evaluated on 23 patients who underwent USguided HDR prostrate brachytherapy with 339 needles being tested in total. Our method detected 98% of the needles with 0.0911±0.0427 mm shaft error and 0.3303±0.3625 mm tip error. Compared with only using mask R-CNN and only using LMMask R-CNN, the proposed method gains a significant improvement of accuracy on both shaft and tip localization. The proposed method automatically digitizes needles per patient with in a second. It streamlines the workflow of USguided HDR prostate brachytherapy and paves the way for the development of real-time treatment planning system that is expected to further elevate the quality and outcome of HDR prostate brachytherapy.

[1]  Hans-Peter Kriegel,et al.  DBSCAN Revisited, Revisited , 2017, ACM Trans. Database Syst..

[2]  Hossein Mobahi,et al.  Large Margin Deep Networks for Classification , 2018, NeurIPS.

[3]  Yang Lei,et al.  Brain tumor segmentation using 3D Mask R-CNN for dynamic susceptibility contrast enhanced perfusion imaging , 2020, Biomedical Applications in Molecular, Structural, and Functional Imaging.

[4]  Yabo Fu,et al.  Deep Learning in Medical Image Registration: A Review , 2020, Physics in medicine and biology.

[5]  Xi Li,et al.  Efficient linear, decoupled, and unconditionally stable scheme for a ternary Cahn-Hilliard type Nakazawa-Ohta phase-field model for tri-block copolymers , 2021, Appl. Math. Comput..

[6]  Yang Lei,et al.  Ultrasound prostate segmentation based on multidirectional deeply supervised V-Net. , 2019, Medical physics.

[7]  Yang Lei,et al.  Multi-needle Localization with Attention U-Net in US-guided HDR Prostate Brachytherapy. , 2020, Medical physics.

[8]  Yi Shen,et al.  Evaluation and comparison of current biopsy needle localization and tracking methods using 3D ultrasound , 2017, Ultrasonics.

[9]  Yang Lei,et al.  Multi-Needle Detection in 3D Ultrasound Images Using Unsupervised Order-Graph Regularized Sparse Dictionary Learning , 2020, IEEE Transactions on Medical Imaging.

[10]  MicroRNAs as potential biomarkers for the diagnosis of traumatic brain injury: a systematic review and meta-analysis , 2019 .

[11]  Yang Lei,et al.  LungRegNet: an unsupervised deformable image registration method for 4D-CT lung. , 2020, Medical physics.

[12]  Xiaofeng Yang,et al.  Improved prostate delineation in prostate HDR brachytherapy with TRUS‐CT deformable registration technology: A pilot study with MRI validation , 2017, Journal of applied clinical medical physics.

[13]  Xiangyang Tang,et al.  Principal component analysis in projection and image domains Another form of spectral imaging in photon-counting CT. , 2020, IEEE transactions on bio-medical engineering.

[14]  Tian Liu,et al.  3D transrectal ultrasound (TRUS) prostate segmentation based on optimal feature learning framework , 2016, SPIE Medical Imaging.

[15]  Jiwoong Jeong,et al.  Multi-needle detection in 3D ultrasound images with sparse dictionary learning , 2020, Medical Imaging.

[16]  Yang Lei,et al.  Automatic Multi-Catheter Detection using Deeply Supervised Convolutional Neural Network in MRI-guided HDR Prostate Brachytherapy. , 2020, Medical physics.

[17]  Yang Lei,et al.  Breast Tumor Segmentation in 3D Automatic Breast Ultrasound Using Mask Scoring R-CNN. , 2020, Medical physics.

[18]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[19]  Peter H. N. de With,et al.  Robust and semantic needle detection in 3D ultrasound using orthogonal-plane convolutional neural networks , 2018, International Journal of Computer Assisted Radiology and Surgery.

[20]  Yang Lei,et al.  4D-CT deformable image registration using multiscale unsupervised deep learning , 2020, Physics in medicine and biology.

[21]  Yang Lei,et al.  Automatic multi-needle localization in ultrasound images using large margin mask RCNN for ultrasound-guided prostate brachytherapy , 2020, Physics in medicine and biology.

[22]  Yang Lei,et al.  Weakly supervised multi-needle detection in 3D ultrasound images with bidirectional convolutional sparse coding , 2020, Medical Imaging.

[23]  Xiaofeng Yang,et al.  Prostate CT segmentation method based on nonrigid registration in ultrasound-guided CT-based HDR prostate brachytherapy. , 2014, Medical physics.

[24]  Yabo Fu,et al.  Deep Learning in Multi-organ Segmentation , 2020, ArXiv.