An adaptive algorithm for detection of multiple-type, positively stained nuclei in IHC images with minimal prior information: application to OLIG2 staining gliomas

We propose a method to detect and segment the oligodendrocytes and gliomas in OLIG2 immunoperoxidase stained tissue sections. Segmentation of cell nuclei is essential for automatic, fast, accurate and consistent analysis of pathology images. In general, glioma cells and oligodendrocytes mostly differ in shape and size within the tissue slide. In OLIG2 stained tissue images, gliomas are represented with irregularly shaped nuclei with varying sizes and brown shades. On the other hand, oligodendrocytes have more regular round nuclei shapes and are smaller in size when compared to glioma cells found in oligodendroglioma, astrocytomas, or oligoastrocytomas. The first task is to detect the OLIG2 positive cell regions within a region of interest image selected from a whole slide. The second task is to segment each cell nucleus and count the number of cell nuclei. However, the cell nuclei belonging to glioma cases have particularly irregular nuclei shapes and form cell clusters by touching or overlapping with each other. In addition to this clustered structure, the shading of the brown stain and the texture of the nuclei differ slightly within a tissue image. The final step of the algorithm is to classify glioma cells versus oligodendrocytes. Our method starts with color segmentation to detect positively stained cells followed by the classification of single individual cells and cell clusters by K-means clustering. Detected cell clusters are segmented with the H-minima based watershed algorithm. The novel aspects of our work are: 1) the detection and segmentation of multiple-type, positively-stained nuclei by incorporating only minimal prior information; and 2) adaptively determining clustering parameters to adjust to the natural variation in staining as well as the underlying cellular structure while accommodating multiple cell types in the image. Performance of the algorithm to detect individual cells is evaluated by sensitivity and precision metrics. Promising segmentation results (91% sensitivity and 86% precision) were achieved for a dataset of fourteen tissue slides with ground truth markings by two pathologists.