Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision

Abstract Dual-arm cooperation is considered as an available approach to improve the poor efficiency by autonomous robotic harvesting. While, cooperating arm movements using visual information is a key challenge for harvesting robots working in non-structured environments. In this paper, we develop a dual-arm cooperative approach for a tomato harvesting robot using a binocular vision sensor. Firstly, a tomato detection algorithm combining AdaBoost classifier and color analysis is proposed and employed by the harvesting robot. Then, a fast three-dimensional scene reconstruction method is obtained in the simulation environment by using point clouds acquired from a stereo camera. Integration of tomato detection, target localization, motion planning and real-time control for dual-arm movements, the dual-arm cooperation for robotic harvesting can be implemented. To validate the proposed approach, field experiments were conducted with the potted tomatoes in greenhouse. Over 96% of target tomatoes were correctly detected with the speed of about 10 fps. The positioning error of robot end-point of less than 10 mm was achieved for large scale direct positioning of the harvesting robot. With the vacuum cup grasping and wide-range cutting, the success rate of robotic harvesting achieved 87.5%. Meanwhile, the harvesting cycle time excluding cruise time was less than 30 s. These results indicate that the dual-arm cooperative approach is feasible and practical for robotic harvesting in non-structured environments.