Automatic Nuclear Segmentation Using Multiscale Radial Line Scanning With Dynamic Programming

In the diagnosis of various cancers by analyzing histological images, automatic nuclear segmentation is an important step. However, nuclear segmentation is a difficult problem because of overlapping nuclei, inhomogeneous staining, and presence of noisy pixels and other tissue components. In this paper, we present an automatic technique for nuclear segmentation in skin histological images. The proposed technique first applies a bank of generalized Laplacian of Gaussian kernels to detect nuclear seeds. Based on the detected nuclear seeds, a multiscale radial line scanning method combined with dynamic programming is applied to extract a set of candidate nuclear boundaries. The gradient, intensity, and shape information are then integrated to determine the optimal boundary for each nucleus in the image. Nuclear overlap limitation is finally imposed based on a Dice coefficient measure such that the obtained nuclear contours do not severely intersect with each other. Experiments have been thoroughly performed on two datasets with H&E and Ki-67 stained images, which show that the proposed technique is superior to conventional schemes of nuclear segmentation.