Integrating deep convolutional neural networks with marker-controlled watershed for overlapping nuclei segmentation in histopathology images

Abstract Nuclei segmentation in histopathology images plays a crucial role in the morphological quantitative analysis of tissue structure and has become a hot research topic. Though numerous efforts have been tried in this research area, the overlapping and touching nuclei segmentation remains a challenging problem. In this paper, we present a novel and effective instance segmentation method for tackling this challenge by integrating Deep Convolutional Neural Networks with Marker-controlled Watershed. Firstly, we design a novel network architecture with multiple segmentation tasks, called Deep Interval-Marker-Aware Network, for learning the foreground, marker, and interval of nuclei, simultaneously. Then the learned interval between overlapping nuclei is used to refine the foreground result of nuclei by using the logical operators. Finally, the learned marker result and the nuclei segmentation result refined by interval are transmitted into the Marker-controlled Watershed for splitting the touching nuclei. The experiments on the standard public datasets demonstrate that our method achieves a substantial improvement compared with state-of-the-art methods. Source codes are available at https://github.com/appiek/Nuclei_Segmentation_Experiments_Demo .