Butterfly-Net: Spatial-Temporal Architecture For Medical Image Segmentation

Radiation therapy tries to maximize the effect of radiation on the tumor and minimize its influence on adjacent tissues. However, it highly depends on the accuracy of the tumor segmentation on the planning radiography images. Tumor contouring today is carried out exclusively with the significant contribution of medical specialists which is a tedious, time-demanding, and expensive task. Further, it is prone to the inter-/intra-observer variation that can affect the reliability of the outcome. Existing methods for automatic tumor segmentation can reduce the influence of these factors, but are not completely reliable and leave a lot of room for improvement. In this work, we exploit Spatio-temporal information from the longitudinal CT scans to improve the deep neural network for tumor segmentation. For this purpose, we devise a novel volumetric Spatio-temporal memory network, Butterfly-Net, which stores the previous scan information and reads for the segmentation at the target time point. Moreover, the effect of clinical factors is investigated in the framework of our volumetric Spatio-temporal memory network. Experimental results on our longitudinal CT scans show that our model could effectively utilize temporal information and clinical factors for tumor segmentation. The code is made publicly available 1

[1]  Jinzhong Yang,et al.  Lung tumor segmentation methods: Impact on the uncertainty of radiomics features for non-small cell lung cancer , 2018, PloS one.

[2]  Ning Xu,et al.  Video Object Segmentation Using Space-Time Memory Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[3]  Guido Gerig,et al.  Fully convolutional structured LSTM networks for joint 4D medical image segmentation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[4]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[5]  Xiangjian He,et al.  Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges , 2019, Journal of Digital Imaging.

[6]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[7]  Ronald M. Summers,et al.  Spatial-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data , 2019, ArXiv.

[8]  Hao Zheng,et al.  Accurate breast lesion segmentation by exploiting spatio-temporal information with deep recurrent and convolutional network , 2019, J. Ambient Intell. Humaniz. Comput..

[9]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[10]  Carlos E Cardenas,et al.  Advances in Auto-Segmentation. , 2019, Seminars in radiation oncology.

[11]  Pengfei Xiong,et al.  Enhanced Memory Network for Video Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[12]  Nassir Navab,et al.  Domain and Geometry Agnostic CNNs for Left Atrium Segmentation in 3D Ultrasound , 2018, MICCAI.

[13]  C. Belka,et al.  Contrast-enhanced, conebeam CT-based, fractionated radiotherapy and follow-up monitoring of orthotopic mouse glioblastoma: a proof-of-concept study , 2020, Radiation Oncology.

[14]  et al.,et al.  Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge , 2018, ArXiv.

[15]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..