Complementary chemometrics and deep learning for semantic segmentation of tall and wide visible and near-infrared spectral images of plants

Close range spectra imaging of agricultural plants is widely performed to support digital plant phenotyping, a task where physicochemical changes in plants are monitored in a non-destructive way. A major step before analyzing the spectral images of plants is to distinguish the plant from the background. Usually, this is an easy task and can be performed using mathematical operations on the combinations of selected spectral bands, such as estimating the normalized difference vegetative index (NDVI). However, when the background of plants contains objects with similar spectral properties as plant then the segmentation based on the threshold of NDVI images can suffer. Another common approach is to train pixel classifiers on spectra extracted from selected locations in the spectral image, but such an approach does not take the spatial information about the plant structure into account. From a technical perspective, plant spectral imaging for digital phenotyping applications usually involves imaging several plants together for a comparative purpose, hence, the imaging scene is relatively big in terms of memory. To solve the challenge of plant segmentation and handling the memory challenge, this study proposes a novel approach, which combines chemometrics with advanced deep learning (DL) based semantic segmentation. The approach has four key steps. As a first step, the spectral image is pre-processed to reduce illumination effects present in the close-range spectral images of plants resulting from the interaction of light with complex plant geometry. Different chemometric pre-processing methods were explored to find possible improvements in the segmentation performance of the DL model. The second step was to perform a principal components analysis (PCA) to reduce the dimensionality of the images, thus drastically reducing their size so that they can be handled more easily using the available computer memory during the training of the DL model. As the third step, small random images (128 × 128) were subsampled from the tall and wide image matrices to generate the training and validation sets for training the DL models. In the last step, a U-net based deep semantic segmentation model was trained and validated on the sub-sampled spectral images. The results showed that the proposed approach allowed efficient handling and training of the DL segmentation model. The intersection over union (IoU) scores for the segmentation was 0.96 for the independent test set image. The segmentation based on variable sorting for normalization and standard normal variate pre-processed data achieved the highest IoU scores. A combination of chemometrics and DL led to an efficient segmentation of tall and wide spectral images which otherwise would have given out-of-memory errors. The developed method can facilitate digital phenotyping tasks where close-range spectral imaging is used to estimate the physicochemical properties of plants. * Corresponding author. E-mail address: puneet.mishra@wur.nl (P. Mishra).