Going to Extremes: Weakly Supervised Medical Image Segmentation

Medical image annotation is a major hurdle for developing precise and robust machine learning models. Annotation is expensive, time-consuming, and often requires expert knowledge, particularly in the medical field. Here, we suggest using minimal user interaction in the form of extreme point clicks to train a segmentation model which, in effect, can be used to speed up medical image annotation. An initial segmentation is generated based on the extreme points utilizing the random walker algorithm. This initial segmentation is then used as a noisy supervision signal to train a fully convolutional network that can segment the organ of interest, based on the provided user clicks. Through experimentation on several medical imaging datasets, we show that the predictions of the network can be refined using several rounds of training with the prediction from the same weakly annotated data. Further improvements are shown utilizing the clicked points within a custom-designed loss and attention mechanism. Our approach has the potential to speed up the process of generating new training datasets for the development of new machine learning and deep learning-based models for, but not exclusively, medical image analysis.

[1]  Ronald M. Summers,et al.  Spatial aggregation of holistically‐nested convolutional neural networks for automated pancreas localization and segmentation☆ , 2017, Medical Image Anal..

[2]  Ender Konukoglu,et al.  Learning to Segment Medical Images with Scribble-Supervision Alone , 2018, DLMIA/ML-CDS@MICCAI.

[3]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[4]  Yaozong Gao,et al.  ASDNet: Attention Based Semi-supervised Deep Networks for Medical Image Segmentation , 2018, MICCAI.

[5]  Henry Medeiros,et al.  Semantic Segmentation Refinement by Monte Carlo Region Growing of High Confidence Detections , 2018, ACCV.

[6]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[7]  Kaiming He,et al.  Group Normalization , 2018, ECCV.

[8]  Mathias Unberath,et al.  UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model , 2017, VCBM.

[9]  Ronald M. Summers,et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms , 2019, ArXiv.

[10]  Max A. Viergever,et al.  Interactive shape models , 2003, SPIE Medical Imaging.

[11]  Eric Granger,et al.  Constrained‐CNN losses for weakly supervised segmentation☆ , 2018, Medical Image Anal..

[12]  Sanja Fidler,et al.  Fast Interactive Object Annotation With Curve-GCN , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Ling Shao,et al.  Extreme Points Derived Confidence Map as a Cue for Class-Agnostic Interactive Segmentation Using Deep Neural Network , 2019, MICCAI.

[14]  Sébastien Ourselin,et al.  DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[16]  Sébastien Ourselin,et al.  Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning , 2017, IEEE Transactions on Medical Imaging.

[17]  Qiong Yang,et al.  Accuracy and efficiency of computer-aided anatomical analysis using 3D visualization software based on semi-automated and automated segmentations. , 2017, Annals of anatomy = Anatomischer Anzeiger : official organ of the Anatomische Gesellschaft.

[18]  Youbao Tang,et al.  Accurate Weakly-Supervised Deep Lesion Segmentation using Large-Scale Clinical Annotations: Slice-Propagated 3D Mask Generation from 2D RECIST , 2018, MICCAI.

[19]  Luc Van Gool,et al.  Deep Extreme Cut: From Extreme Points to Object Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  Ronald M. Summers,et al.  Self-learning to detect and segment cysts in lung CT images without manual annotation , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[21]  Andriy Myronenko,et al.  3D MRI brain tumor segmentation using autoencoder regularization , 2018, BrainLes@MICCAI.

[22]  Daguang Xu,et al.  3D Anisotropic Hybrid Network: Transferring Convolutional Features from 2D Images to 3D Anisotropic Volumes , 2017, MICCAI.

[23]  BoykovYuri,et al.  Graph Cuts and Efficient N-D Image Segmentation , 2006 .

[24]  Henry Medeiros,et al.  FreeLabel: A Publicly Available Annotation Tool Based on Freehand Traces , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[25]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[26]  B. van Ginneken,et al.  Use of Volumetry for Lung Nodule Management: Theory and Practice. , 2017, Radiology.

[27]  Ronald M. Summers,et al.  DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation , 2015, MICCAI.

[28]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[29]  Geoff Dougherty,et al.  Medical Image Processing: Techniques and Applications , 2011 .

[30]  Dean C. Barratt,et al.  Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks , 2018, IEEE Transactions on Medical Imaging.

[31]  Sébastien Ourselin,et al.  Slic-Seg: A minimally interactive segmentation of the placenta from sparse and motion-corrupted fetal MRI in multiple views , 2016, Medical Image Anal..

[32]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[33]  I. Wolf,et al.  3D heart segmentation and volumetry using deformable shape models , 2007, 2007 Computers in Cardiology.

[34]  Daguang Xu,et al.  Weakly supervised segmentation from extreme points , 2019, LABELS/HAL-MICCAI/CuRIOUS@MICCAI.

[35]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Nima Tajbakhsh,et al.  Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation , 2019, Medical Image Anal..

[37]  Fred A. Hamprecht,et al.  End-To-End Learned Random Walker for Seeded Image Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Amir Alansary,et al.  Learning under Distributed Weak Supervision , 2016, ArXiv.

[39]  Yu Liu,et al.  A review of semantic segmentation using deep neural networks , 2017, International Journal of Multimedia Information Retrieval.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Angela Yao,et al.  Content-Aware Multi-Level Guidance for Interactive Instance Segmentation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Frank Keller,et al.  Extreme Clicking for Efficient Object Annotation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Daguang Xu,et al.  Interactive segmentation of medical images through fully convolutional neural networks , 2019, ArXiv.