Reconstruction of kiwifruit fruit geometry using a CGAN trained on a synthetic dataset

Abstract Non-destructive crop yield estimation is a major ambition for the development of digital horticulture systems, where the goal is to have a machine that can look at fruit on the vine/tree and estimate key performance metrics such as fruit weight, size and shape. The on-orchard partial occlusion problem is a major obstacle for the creation of such systems, where it is not possible for a single camera/scanner to capture the complete fruit surface due to its limited field of view. In this work, a deep learning approach was taken to solve this problem. Framing the issue as an image-to-image translation problem, a Conditional Generative Adversarial Network was trained to realistically reconstruct the enclosed and complete surface of kiwifruit when supplied with incomplete surface data, where the surface data was missing due to occlusion. Rather than training the deep learning algorithm with empirically collected data, an alternative approach was taken: the model was trained using a synthesised dataset, a large collection of kiwifruit shapes generated computationally via a Monte-Carlo routine. This was an attempt to generalise the approach to be applicable to other crops and other domains, and to provide substantial savings in time, labour and material costs. The trained model was later applied to a smaller population of kiwifruit empirically scanned using an infrared scanner and could predict fruit weight with a mean absolute percentage error of less than 5% and was successful in realistically reconstructing the whole enclosed surface over a range of sizes, shapes and orientations.