Detection of red and bicoloured apples on tree with an RGB-D camera

Recognising and accurately locating fruits on a tree is a critical challenge in developing fruit-by-fruit robotic harvesting. Many researchers have investigated the potential of red, green, blue (RGB) colour imaging for this purpose, but have had limited success due to the occlusion of the target fruits by foliage, branches or other fruits as well as due to the non-uniform and unstructured nature of an orchard environment. Recently, novel, cost-effective camera systems have become available which provide both colour (RGB) and three dimensional (3D) shape information. As these have shown potential for 3D perception for robots operating in unstructured environments, the potential of such an RGB-D camera for the detection and localisation of red and bicoloured apples on tree was investigated in this study. Images were acquired with this camera system in fruit orchards under a light shield blocking direct sunlight, and an algorithm to detect and localise red and bicoloured apples based on colour and shape features was developed. When the algorithm was applied to the data acquired in these orchards, 100% of the fully visible apples and 82% of the partially occluded apples were detected correctly. The location estimation error was below 10 mm in all the coordinate axes of the Cartesian space. This high detection and location accuracy and short processing time (below 1 s for simultaneous detection of 20 apples), makes the developed algorithm suitable for implementation in a robotic harvesting system, and for yield estimation and orchard monitoring.