User-Guided Domain Adaptation for Rapid Annotation from User Interactions: A Study on Pathological Liver Segmentation

Mask-based annotation of medical images, especially for 3D data, is a bottleneck in developing reliable machine learning models. Using minimal-labor (UIs) to guide the annotation is promising, but challenges remain on best harmonizing the mask prediction with the UIs. To address this, we propose the user-guided domain adaptation (UGDA) framework, which uses prediction-based adversarial domain adaptation (PADA) to model the combined distribution of UIs and mask predictions. The UIs are then used as anchors to guide and align the mask prediction. Importantly, UGDA can both learn from unlabelled data and also model the high-level semantic meaning behind different UIs. We test UGDA on annotating pathological livers using a clinically comprehensive dataset of 927 patient studies. Using only extreme-point UIs, we achieve a mean (worst-case) performance of \(96.1\%\) (\(94.9\%\)), compared to \(93.0\%\) (\(87.0\%\)) for deep extreme points (DEXTR). Furthermore, we also show UGDA can retain this state-of-the-art performance even when only seeing a fraction of available UIs, demonstrating an ability for robust and reliable UI-guided segmentation with extremely minimal labor demands.

[1]  Nima Tajbakhsh,et al.  Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation , 2019, Medical Image Anal..

[2]  Daguang Xu,et al.  Weakly supervised segmentation from extreme points , 2019, LABELS/HAL-MICCAI/CuRIOUS@MICCAI.

[3]  Frank Keller,et al.  Extreme Clicking for Efficient Object Annotation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[6]  Fengmao Lv,et al.  Domain Adaptive Semantic Segmentation Through Structure Enhancement , 2018, ECCV Workshops.

[7]  Adam P. Harrison,et al.  Co-Heterogeneous and Adaptive Segmentation from Multi-Source and Multi-Phase CT Imaging Data: A Study on Pathological Liver and Lesion Segmentation , 2020, ECCV.

[8]  Ming-Hsuan Yang,et al.  Learning to Adapt Structured Output Space for Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Sébastien Ourselin,et al.  DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Chi-Wing Fu,et al.  H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation From CT Volumes , 2018, IEEE Transactions on Medical Imaging.

[11]  Adam P. Harrison,et al.  IntellEditS: Intelligent Learning-Based Editor of Segmentations , 2013, MICCAI.

[12]  Hao Chen,et al.  The Liver Tumor Segmentation Benchmark (LiTS) , 2019, Medical Image Anal..

[13]  Nuno Vasconcelos,et al.  Bidirectional Learning for Domain Adaptation of Semantic Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  U. Neumann,et al.  Fast and accurate liver volumetry prior to hepatectomy. , 2016, HPB : the official journal of the International Hepato Pancreato Biliary Association.

[15]  Xingyi Zhou,et al.  Bottom-Up Object Detection by Grouping Extreme and Center Points , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Hao Chen,et al.  Light-Weight Hybrid Convolutional Network for Liver Tumor Segmentation , 2019, IJCAI.

[17]  Martin Styner,et al.  Comparison and Evaluation of Methods for Liver Segmentation From CT Datasets , 2009, IEEE Transactions on Medical Imaging.

[18]  Klaus H. Maier-Hein,et al.  nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation , 2018, Bildverarbeitung für die Medizin.

[19]  Ronald M. Summers,et al.  Progressive and Multi-path Holistically Nested Neural Networks for Pathological Lung Segmentation from CT Images , 2017, MICCAI.

[20]  Kunio Doi,et al.  Automated hepatic volumetry for living related liver transplantation at multisection CT. , 2006, Radiology.

[21]  Sébastien Ourselin,et al.  Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning , 2017, IEEE Transactions on Medical Imaging.

[22]  Luc Van Gool,et al.  Deep Extreme Cut: From Extreme Points to Object Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Klaus H. Maier-Hein,et al.  Abstract: nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation , 2019, Bildverarbeitung für die Medizin.

[24]  R. Killi,et al.  Donor safety and remnant liver volume in living donor liver transplantation , 2008, Liver transplantation : official publication of the American Association for the Study of Liver Diseases and the International Liver Transplantation Society.

[25]  Jianmin Zheng,et al.  CT volumetry of the liver: where does it stand in clinical practice? , 2014, Clinical radiology.

[26]  Konstantinos Kamnitsas,et al.  DeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks , 2016, IEEE Transactions on Medical Imaging.

[27]  Taesung Park,et al.  CyCADA: Cycle-Consistent Adversarial Domain Adaptation , 2017, ICML.

[28]  Gareth Funka-Lea,et al.  An Energy Minimization Approach to the Data Driven Editing of Presegmented Images/Volumes , 2006, MICCAI.

[29]  Ender Konukoglu,et al.  Learning to Segment Medical Images with Scribble-Supervision Alone , 2018, DLMIA/ML-CDS@MICCAI.