Segmentation of Nuclei From 3D Microscopy Images of Tissue via Graphcut Optimization

Nuclear segmentation in 3D microscopic tissue images remains a challenge due to tight packing of the cells, low contrast and poor depth resolution. To address this problem, we developed a robust and accurate nuclear segmentation algorithm suited for tissue samples of higher organisms such as mice and humans. The study was inspired by previous works in graphcut-based segmentation in areas other than optical microscopy. We propose a novel seed detection method for nuclei in 3D tissue images that uses a robust model-based 2D slice by slice segmentation followed by a non-maximal suppression like algorithm for selection of the most prospective set of seeds. After seed detection, for each target nucleus the method transformed the microscopic volume to a geometric volume in spherical space with respect to the respective seed point and found the globally optimal surface in that geometric volume which separated the target cell nucleus from the rest of the volume using a graphcut-based algorithm. Comparing the automatic segmentation maps obtained by the proposed method to those obtained from a number of state-of-the art methods demonstrate superior robustness and accuracy of the proposed method in terms of three evaluation metrics. Along with the automatic method, we also present an interactive version of the algorithm.