Detecting and segmenting cell nuclei in two-dimensional microscopy images

Introduction: Cell nuclei are important indicators of cellular processes and diseases. Segmentation is an essential stage in systems for quantitative analysis of nuclei extracted from microscopy images. Given the wide variety of nuclei appearance in different organs and staining procedures, a plethora of methods have been described in the literature to improve the segmentation accuracy and robustness. Materials and Methods: In this paper, we propose an unsupervised method for cell nuclei detection and segmentation in two-dimensional microscopy images. The nuclei in the image are detected automatically using a matching-based method. Next, edge maps are generated at multiple image blurring levels followed by edge selection performed in polar space. The nuclei contours are refined iteratively in the constructed edge pyramid. The validation study was conducted over two cell nuclei datasets with manual labeling, including 25 hematoxylin and eosin-stained liver histopathology images and 35 Papanicolaou-stained thyroid images. Results: The nuclei detection accuracy was measured by miss rate, and the segmentation accuracy was evaluated by two types of error metrics. Overall, the nuclei detection efficiency of the proposed method is similar to the supervised template matching method. In comparison to four existing state-of-the-art segmentation methods, the proposed method performed the best with average segmentation error 10.34% and 0.33 measured by area error rate and normalized sum of distances (×10). Conclusion: Quantitative analysis showed that the method is automatic and accurate when segmenting cell nuclei from microscopy images with noisy background and has the potential to be used in clinic settings.