Efficient Nucleus Detection in Digital Pathology Images using Multi-task and Multi-scale Instance Segmentation Network

Automatic cell spotting is a challenging problem in biomedical image analysis. Efficient detection of the nuclei in these cells is important for both clinicians and researchers. Technically, this problem suffers from background clutters of the pathological tissue and has difficulty in tiny objects recognition. To solve these, we tend to apply deep learning based method and treat this as a nucleus instance segmentation task. Structurally, we modify the widely used Mask R-CNN structure in region proposal networks with multi-scale, resulting in the aware of tiny objects. And for adapting to our specific task, we take advantage of the “anchor” mechanism. Compared with traditional detection methods, the learned features are more robust to massive tissue backgrounds. Moreover, our model can simultaneously detect and segment the nucleus through an end to end training strategy, which of great application values. To our best knowledge, this work is the first time to the nucleus detection task. Experimentally, we test our method on the dataset of 2018 Data Science Bowl. As a result, both visualization and quantization indicators show the effectiveness of our method.