TwinLiverNet: Predicting TACE Treatment Outcome from CT scans for Hepatocellular Carcinoma using Deep Capsule Networks

Predicting response to treatment plays a key role to assist radiologists in hepato-cellular carcinoma (HCC) therapy planning. The most widely used treatment for unresectable HCC is the trans-arterial chemoembolization (TACE). A complete radiological response after the first TACE is a reliable predictor of treatment favourable outcome. However, visual inspection of contrast-enhanced CT scans is time-consuming, error prone and too operator-dependent. Thus, in this paper we propose TwinLiverNet: a deep neural network that is able to predict TACE treatment outcome through learning visual cue from CT scans. TwinLiverNet, specifically, integrates 3D convolutions and capsule networks and is designed to process simultaneously late arterial and delayed phases from contrast-enhanced CTs. Experimental results carried out on a dataset consisting of 126 HCC lesions show that TwinLiverNet reaches an average accuracy of 82% in predicting complete response to TACE treatment. Furthermore, combining multiple CT phases (specifically, late arterial and delayed ones) yields a performance increase of over 12 percent points. Finally, the introduction of capsule layers into the model avoids the model to overfit, while enhancing accuracy.Clinical relevance— TwinLiverNet supports radiologists in visual inspection of CT scans to assess TACE treatment outcome, while reducing inter-operator variability.

[1]  Geoffrey E. Hinton,et al.  Transforming Auto-Encoders , 2011, ICANN.

[2]  María J. Ledesma-Carbayo,et al.  Optimal multiresolution 3D level-set method for liver segmentation incorporating local curvature constraints , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[3]  M. Kudo,et al.  Global patterns of hepatocellular carcinoma management from diagnosis to death: the BRIDGE Study , 2015, Liver international : official journal of the International Association for the Study of the Liver.

[4]  Hien Van Nguyen,et al.  Fast CapsNet for Lung Cancer Screening , 2018, MICCAI.

[5]  Sebastian J. Schlecht,et al.  SurvivalNet: Predicting patient survival from diffusion weighted magnetic resonance images using cascaded fully convolutional and 3D Convolutional Neural Networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[6]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[7]  Zhiping Lin,et al.  Random feature subspace ensemble based Extreme Learning Machine for liver tumor detection and segmentation , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[8]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[9]  Myeong-Jin Kim,et al.  Complete response at first chemoembolization is still the most robust predictor for favorable outcome in hepatocellular carcinoma. , 2015, Journal of hepatology.

[10]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[11]  Xiao Han,et al.  Automatic Liver Lesion Segmentation Using A Deep Convolutional Neural Network Method , 2017, ArXiv.

[12]  H. Peitgen,et al.  Segmentation of Liver Metastases in CT Scans by Adaptive Thresholding and Morphological Processing , 2008, The MIDAS Journal.

[13]  Eun Sun Lee,et al.  Prediction of Therapeutic Response of Hepatocellular Carcinoma to Transcatheter Arterial Chemoembolization Based on Pretherapeutic Dynamic CT and Textural Findings. , 2017, AJR. American journal of roentgenology.

[14]  Wei Zhao,et al.  Residual convolutional neural network for predicting response of transarterial chemoembolization in hepatocellular carcinoma from CT imaging , 2019, European Radiology.

[15]  Ryanne A. Brown,et al.  Impact of a deep learning assistant on the histopathologic classification of liver cancer , 2020, npj Digital Medicine.

[16]  Samuel Kadoury,et al.  Metastatic Liver Tumor Segmentation Using Texture-Based Omni-Directional Deformable Surface Models , 2014, ABDI@MICCAI.

[17]  Johan Montagnat,et al.  Fully Automatic Anatomical, Pathological, and Functional Segmentation from CT Scans for Hepatic Surgery , 2000, Medical Imaging: Image Processing.

[18]  Kumardeep Chaudhary,et al.  Deep Learning–Based Multi-Omics Integration Robustly Predicts Survival in Liver Cancer , 2017, Clinical Cancer Research.

[19]  Seyed-Ahmad Ahmadi,et al.  Automatic Liver and Lesion Segmentation in CT Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields , 2016, MICCAI.

[20]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Simone Palazzo,et al.  Interpretable Deep Model For Predicting Gene-Addicted Non-Small-Cell Lung Cancer In Ct Scans , 2021, 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI).

[22]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[23]  Gregory J. Gores,et al.  A global view of hepatocellular carcinoma: trends, risk, prevention and management , 2019, Nature Reviews Gastroenterology & Hepatology.

[24]  Xiaoming Liu,et al.  Automatic segmentation of liver tumors from multiphase contrast-enhanced CT images based on FCNs , 2017, Artif. Intell. Medicine.

[25]  Ulas Bagci,et al.  Capsules for Biomedical Image Segmentation , 2020, Medical Image Anal..

[26]  Te-Wei Ho,et al.  Disease-Free Survival after Hepatic Resection in Hepatocellular Carcinoma Patients: A Prediction Approach Using Artificial Neural Network , 2012, PloS one.

[27]  Pham The Bao,et al.  Liver Tumor Segmentation from MR Images Using 3D Fast Marching Algorithm and Single Hidden Layer Feedforward Neural Network , 2016, BioMed research international.

[28]  V. Mazzaferro,et al.  Evolving strategies for the management of intermediate-stage hepatocellular carcinoma: available evidence and expert opinion on the use of transarterial chemoembolization. , 2011, Cancer treatment reviews.

[29]  Fucang Jia,et al.  Automatic Segmentation of Liver Tumor in CT Images with Deep Convolutional Neural Networks , 2015 .

[30]  Hayit Greenspan,et al.  Fully Convolutional Network for Liver Segmentation and Lesions Detection , 2016, LABELS/DLMIA@MICCAI.