Tracing in 2D to Reduce the Annotation Effort for 3D Deep Delineation

The difficulty of obtaining annotations to build training databases still slows down the adoption of recent deep learning approaches for biomedical image analysis. In this paper, we show that we can train a Deep Net to perform 3D volumetric delineation given only 2D annotations in Maximum Intensity Projections (MIP) of the training volumes. This significantly reduces the annotation time: We conducted a user study that suggests that annotating 2D projections is on average twice as fast as annotating the original 3D volumes. Our technical contribution is a loss function that evaluates a 3D prediction against annotations of 2D projections. It is inspired by space carving, a classical approach to reconstructing complex 3D shapes from arbitrarily-positioned cameras. It can be used to train any deep network with volumetric output, without the need to change the network's architecture. Substituting the loss is all it takes to enable 2D annotations in an existing training setup. In extensive experiments on 3D light microscopy images of neurons and retinal blood vessels, and on Magnetic Resonance Angiography (MRA) brain scans, we show that, when trained on projection annotations, deep delineation networks perform as well as when they are trained using costlier 3D annotations.

[1]  Ender Konukoglu,et al.  Learning to Segment Medical Images with Scribble-Supervision Alone , 2018, DLMIA/ML-CDS@MICCAI.

[2]  Ronan Collobert,et al.  From image-level to pixel-level labeling with Convolutional Networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Raquel Urtasun,et al.  DeepRoadMapper: Extracting Road Topology from Aerial Images , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Yingli Tian,et al.  Coarse-to-Fine Semantic Segmentation From Image-Level Labels , 2018, IEEE Transactions on Image Processing.

[5]  Jan Dirk Wegner,et al.  A Higher-Order CRF Model for Road Network Extraction , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Pascal Fua,et al.  Detecting Irregular Curvilinear Structures in Gray Scale and Color Imagery Using Multi-directional Oriented Flux , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Vincent Lepetit,et al.  Supervised Feature Learning for Curvilinear Structure Segmentation , 2013, MICCAI.

[8]  Bernt Schiele,et al.  Simple Does It: Weakly Supervised Instance and Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Matthew Cook,et al.  Efficient automatic 3D-reconstruction of branching neurons from EM data , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Suha Kwak,et al.  Learning Pixel-Level Semantic Affinity with Image-Level Supervision for Weakly Supervised Semantic Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  H. Sebastian Seung,et al.  Maximin affinity learning of image segmentation , 2009, NIPS.

[12]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[13]  Arthur W Toga,et al.  The role of brain vasculature in neurodegenerative disorders , 2018, Nature Neuroscience.

[14]  Max W. K. Law,et al.  Three Dimensional Curvilinear Structure Detection Using Optimally Oriented Flux , 2008, ECCV.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Konstantinos Kamnitsas,et al.  DeepCut: Object Segmentation From Bounding Box Annotations Using Convolutional Neural Networks , 2016, IEEE Transactions on Medical Imaging.

[17]  J. K. Smith,et al.  Vessel tortuosity and brain tumor malignancy: a blinded study. , 2005, Academic radiology.

[18]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[19]  George Papandreou,et al.  Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation , 2015, ArXiv.

[20]  S. N. Merchant,et al.  MS-Net: Mixed-Supervision Fully-Convolutional Networks for Full-Resolution Segmentation , 2018, MICCAI.

[21]  Pascal Fua,et al.  Learning to Segment 3D Linear Structures Using Only 2D Annotations , 2018, MICCAI.

[22]  Luc Van Gool,et al.  Deep Retinal Image Understanding , 2016, MICCAI.

[23]  Hanchuan Peng,et al.  Automatic tracing of ultra-volumes of neuronal images , 2016, Nature Methods.

[24]  Guido Gerig,et al.  3D Multi-scale line filter for segmentation and visualization of curvilinear structures in medical images , 1997, CVRMed.

[25]  Lin Yang,et al.  Deep Learning Based Instance Segmentation in 3D Biomedical Images Using Weak Annotation , 2018, MICCAI.

[26]  Chandan Singh,et al.  Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Victor S. Lempitsky,et al.  N4-Fields: Neural Network Nearest Neighbor Fields for Image Transforms , 2014, ArXiv.

[28]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[29]  Pascal Fua,et al.  Active Learning and Proofreading for Delineation of Curvilinear Structures , 2017, MICCAI.

[30]  Pascal Fua,et al.  Beyond the Pixel-Wise Loss for Topology-Aware Delineation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Jian Sun,et al.  ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Geoffrey E. Hinton,et al.  Machine Learning for Aerial Image Labeling , 2013 .

[33]  Holger Roth,et al.  Tracking and Segmentation of the Airways in Chest CT Using a Fully Convolutional Network , 2017, MICCAI.

[34]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[35]  Hanchuan Peng,et al.  Virtual finger boosts three-dimensional imaging and microsurgery as well as terabyte volume image visualization and analysis , 2014, Nature Communications.

[36]  Geoffrey E. Hinton,et al.  Learning to Detect Roads in High-Resolution Aerial Images , 2010, ECCV.

[37]  Steven S. Laurie,et al.  Vascular remodeling in pulmonary hypertension , 2013, Journal of Molecular Medicine.

[38]  Joachim Hornegger,et al.  3D annotation and manipulation of medical anatomical structures , 2009, Medical Imaging.

[39]  James G. King,et al.  Reconstruction and Simulation of Neocortical Microcircuitry , 2015, Cell.

[40]  Vincent Lepetit,et al.  Multiscale Centerline Detection , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Shaohua Kevin Zhou,et al.  Hierarchical Discriminative Framework for Detecting Tubular Structures in 3D Images , 2013, IPMI.

[42]  Nico Westerhof,et al.  Vascular narrowing in pulmonary arterial hypertension is heterogeneous: rethinking resistance , 2017, Physiological reports.