Robotic Aubergine Harvesting Using Dual-Arm Manipulation

Interest in agricultural automation has increased considerably in recent decades due to benefits such as improving productivity or reducing the labor force. However, there are some current problems associated with unstructured environments make developing a robotic harvester a challenge. This article presents a dual-arm aubergine harvesting robot consisting of two robotic arms configured in an anthropomorphic manner to optimize the dual workspace. To detect and locate the aubergines automatically, we implemented an algorithm based on a support vector machine (SVM) classifier and designed a planning algorithm for scheduling efficient fruit harvesting that coordinates the two arms throughout the harvesting process. Finally, we propose a novel algorithm for dealing with occlusions using the capabilities of the dual-arm robot for coordinate work. Therefore, the main contribution of this study is the implementation and validation of a dual-arm harvesting robot with planning and control algorithms, which, depending on the locations of the fruits and the configuration of the arms, enables the following: (i) the simultaneous harvesting of two aubergines; (ii) the harvesting of a single aubergine with a single arm; or (iii) a collaborative behavior between the arms to solve occlusions. This cooperative operation mimics complex human harvesting motions such as using one arm to push leaves aside while the other arm picks the fruit. The performance of the proposed harvester is evaluated through laboratory tests that simulate the most common real-world scenarios. The results show that the robotic harvester has a success rate of 91.67% and an average cycle time of 26 s/fruit.

[1]  Martin F. Stoelen,et al.  Soft manipulator robot for selective tomato harvesting , 2019 .

[2]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[3]  Bruce MacDonald,et al.  Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms , 2019, Biosystems Engineering.

[4]  Yadong Liu,et al.  Computer vision technology in agricultural automation —A review , 2020 .

[5]  Yong Zhou,et al.  A survey of semi- and weakly supervised semantic segmentation of images , 2019, Artificial Intelligence Review.

[6]  Pablo González de Santos,et al.  Modular Dual-Arm Robot for Precision Harvesting , 2019, ROBOT.

[7]  Anton Satria Prabuwono,et al.  Automatic image segmentation using sobel operator and k-means clustering: A case study in volume measurement system for food products , 2015, 2015 International Conference on Science in Information Technology (ICSITech).

[8]  Roemi Fernández,et al.  Automatic Detection of Field-Grown Cucumbers for Robotic Harvesting , 2018, IEEE Access.

[9]  Bofeng Zhang,et al.  An Improved Method to Reduce Over-Segmentation of Watershed Transformation and its Application in the Contour Extraction of Brain Image , 2009, 2009 Eighth IEEE International Conference on Dependable, Autonomic and Secure Computing.

[10]  David Kaeli,et al.  Heterogeneous Computing with OpenCL 2.0 , 2015 .

[11]  Alexandre Campeau-Lecours,et al.  Kinova Modular Robot Arms for Service Robotics Applications , 2017, Int. J. Robotics Appl. Technol..

[12]  Jianfei Cai,et al.  Beyond pixels: A comprehensive survey from bottom-up to semantic image segmentation and cosegmentation , 2015, J. Vis. Commun. Image Represent..

[13]  David Kaeli,et al.  Case study: Image clustering , 2015 .

[14]  Zheng Lu,et al.  Survey of weakly supervised semantic segmentation methods , 2018, 2018 Chinese Control And Decision Conference (CCDC).

[15]  Sotiris B. Kotsiantis,et al.  Supervised Machine Learning: A Review of Classification Techniques , 2007, Informatica.

[16]  Digvijay S. Pawar,et al.  Analyzing Gap Acceptance Behavior at Unsignalized Intersections Using Support Vector Machines, Decision Tree and Random Forests , 2017, ANT/SEIT.

[17]  Yu Liu,et al.  A review of semantic segmentation using deep neural networks , 2017, International Journal of Multimedia Information Retrieval.

[18]  Yuanshen Zhao,et al.  A review of key techniques of vision-based control for harvesting robot , 2016, Comput. Electron. Agric..

[19]  Pablo González de Santos,et al.  ROS Framework for Perception and Dual-Arm Manipulation in Unstructured Environments , 2019, ROBOT.

[20]  Stefan Schaal,et al.  STOMP: Stochastic trajectory optimization for motion planning , 2011, 2011 IEEE International Conference on Robotics and Automation.

[21]  James Patrick Underwood,et al.  Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry , 2016, Sensors.

[22]  Tateshi Fujiura,et al.  Cherry-harvesting robot , 2008 .

[23]  Takanori Fukao,et al.  An automated fruit harvesting robot by using deep learning , 2019, ROBOMECH Journal.

[24]  Amira S. Ashour,et al.  Neutrosophic sets in dermoscopic medical image segmentation , 2019, Neutrosophic Set in Medical Image Analysis.

[25]  Xiaochun Cao,et al.  Survey of recent progress in semantic image segmentation with CNNs , 2017, Science China Information Sciences.

[26]  José García Rodríguez,et al.  A survey on deep learning techniques for image and video semantic segmentation , 2018, Appl. Soft Comput..

[27]  Tao Wang,et al.  Dual-arm cooperation and implementing for robotic harvesting tomato using binocular vision , 2019, Robotics Auton. Syst..

[28]  Patrizia Busato,et al.  Machine Learning in Agriculture: A Review , 2018, Sensors.

[29]  Yassine Ruichek,et al.  Survey on semantic segmentation using deep learning techniques , 2019, Neurocomputing.

[30]  H. Yoshida,et al.  DEVELOPMENT OF AN END EFFECTOR FOR A STRAWBERRY-HARVESTING ROBOT , 2008 .

[31]  Wei Zou,et al.  Design and test of robotic harvesting system for cherry tomato , 2018 .

[32]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[33]  Shigehiko Hayashi,et al.  Robotic Harvesting System for Eggplants , 2002 .

[34]  Serge Beucher,et al.  Watershed, Hierarchical Segmentation and Waterfall Algorithm , 1994, ISMM.

[35]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[36]  Moshe P. Mann,et al.  Harvest-order planning for a multiarm robotic harvester , 2014 .

[37]  Wei Sun,et al.  Methods and datasets on semantic segmentation: A review , 2018, Neurocomputing.

[38]  Giorgio Grasso,et al.  Localization of spherical fruits for robotic harvesting , 2001, Machine Vision and Applications.

[39]  Luc Vincent,et al.  Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Wan Ishak Wan Ismail,et al.  Design and development of eggplant harvester for gantry system , 2010 .