Computer aided tool for automatic detection and delineation of nucleus from oral histopathology images for OSCC screening

Abstract Characterization of nucleus from oral tissue histology images is important for oral squamous cell carcinoma (OSCC) diagnosis. Therefore, the accurate delineation of nucleus is the foremost task to initiate the quantification of its various properties such as morphology, texture, intensity, etc. In this paper, we propose a two-stage computer-aided tool for automatic detection (the first stage) and delineation of the detected nucleus (second stage) from oral histological images to aid clinician in OSCC diagnosis. In the first stage, 81 × 81 patches were extracted from our image database followed by the wavelet-based downsampling process to generate 21 × 21 patches, which were used to train the deep convolution neural network (CNN) architecture. This approach was used to automatically detect nucleus from OSCC images and achieved 88.87% recall and 82.03% precision. After detection, 81 × 81 image patches were extracted from those detected nucleus regions and further processed through the second stage of proposed pipeline to segment the exact boundary of nucleus using active contour (Chan-Vese model) after applying edge enhancing nonsubsampled contourlet transform. The proposed segmentation methodology performed well with 94.22% Dice coefficient, 89.38% Jaccard index, 97.56% precision, and 91.58% recall. We view this work as the first attempt on oral tissue histology image computation for joint nucleus detection and segmentation to diagnose OSCC to the best of our knowledge.