Fine-tuning convolutional neural network with transfer learning for semantic segmentation of ground-level oilseed rape images in a field with high weed pressure

Abstract Image processing technology has gained considerable attention in agricultural proximal sensing applications, including plant disease detection, vegetation fraction estimation, monitoring of the crop growth status, and image-based site-specific management. Image segmentation is the first and crucial step to process complex infield images. However, image segmentation by either hand engineered-based or deep learning-based methods that train the entire system from scratch is a daunting task and needs several hundreds of labeled images that may be difficult to obtain in practice. The recent development of transfer learning has shown the potential of transferring the learned feature detectors of a pre-trained convolutional neural network to a new image dataset. This study was thus aimed to evaluate three transfer learning methods using a VGG16-based encoder net for semantic segmentation of oilseed rapes images in a field with high-density weeds. Three different transfer learning approaches using a VGG16-based encoder model were proposed, and their performances were compared to a VGG19-based encoder net. Relying on the intensive use of data augmentation and transfer learning, we showed that such networks could be trained end-to-end using a few annotated training images. The highest accuracy of 96% was obtained by the VGG16-based encoder net in which the fine-tuned model was only used for feature extraction and the segmentation was performed using shallow machine learning classifiers (MLCs). Transfer learning demonstrated to be efficient and presented a robust performance in segmenting plants amongst high-density weeds. The implementation of MLCs is reasonable for real-time applications with the segmentation time less than 0.05 s/image.