Residual Attention Generative Adversarial Networks for Nuclei Detection on Routine Colon Cancer Histology Images

The automatic detection of nuclei in pathological images plays an important role in diagnosis and prognosis of cancers. Most nuclei detection algorithms are based on the assumption that the nuclei center should have larger responses than their surroundings in the probability map of the pathological image, which in turn transforms the detection or localization problem into finding the local maxima on the probability map. However, all the existing studies used regression algorithms to determine the probability map, which neglect to take the spatial contiguity within the probability map into consideration. In order to capture the higher-order consistency within the generated probability map, we propose an approach called Residual Attention Generative Adversarial Network (i.e., RAGAN) for nuclei detection. Specifically, the objective function of the RAGAN model combines a detection term with an adversarial term. The adversarial term adopts a generator called Residual Attention U-Net (i.e., RAU-Net) to produce the probability maps that cannot be distinguished by the ground-truth. Based on the adversarial model, we can simultaneously estimate the probabilities of many pixels with high-order consistency, by which we can derive a more accurate probability map. We evaluate our method on a public colorectal adenocarcinoma images dataset with 29756 nuclei. Experimental results show that our method can achieve the F1 Score of 0.847 (with a Precision of 0.859 and a Recall of 0.836) for nuclei detection, which is superior to the conventional methods.