Joint Plant and Leaf Instance Segmentation on Field-Scale UAV Imagery

Monitoring of fields and breeding plots is critical for farmers, plant scientists, and breeders. In this process, a key objective is to assess and monitor the growth stages together with the number of individual plants on the field. Traditionally, this in-field assessment is performed manually and thus is limited in temporal and spatial throughput. In contrast, vision-based systems offer the potential to assess these traits frequently in an automated fashion on a large scale. The primary target of these systems is to detect and segment each plant and its leaves since this information directly correlates to the growth stage and allows for detailed monitoring. In this paper, we address the problem of automated, instance-level plant monitoring in agricultural fields and breeding plots. We propose a vision-based approach to perform a joint instance segmentation of crop plants and leaves in breeding plots. We develop a convolutional neural network to determine the position of specific plant keypoints and group pixels to detect individual leaf and plant instances. Finally, we provide a pixel-wise instance segmentation of each crop and its associated leaves based on orthorectified RGB images captured by UAVs. The experimental evaluation shows that our method outperforms state-of-the-art instance segmentation approaches such as Mask-RCNN on this task.