Immature green citrus fruit detection using color and thermal images

Abstract Citrus fruit detection is one of the most important and challenging steps in citrus yield mapping. The distinct color differences between the ripe fruit and leaves allowed previously-described imaging-based methods to achieve good results. However, immature green citrus fruit detection, which aims to provide valuable information for citrus yield mapping at earlier stages is much more difficult because the fruit and leaf colors are very similar. This study combines color and thermal images for immature green fruit detections. Experiments identified optimal conditions for thermal imaging. A multimodal imaging platform was built to integrate color and thermal cameras. A novel image registration method was developed for combining color and thermal images and matching fruit in both images which achieved pixel-level accuracy. A new Color-Thermal Combined Probability (CTCP) algorithm was created to effectively fuse information from the color and thermal images to classify potential image regions into fruit and non-fruit classes. Algorithms were also developed to integrate image registration, information fusion and fruit classification and detection into a single step for real-time processing. An increase in recall rate from 78.1% when using only color images to 90.4% after fusing the color and thermal images was obtained at similar precision rates, and an increase in precision rate from 86.6% to 95.5% was obtained at similar recall rates. The fusion of the color and thermal images effectively improved immature green citrus fruit detection.