Object Localization and Depth Estimation for Eye-in-Hand Manipulator Using Mono Camera

This paper proposes the object localization and depth estimation to select and set goals for robots via machine vision. An algorithm based on a deep region-based convolution neural network (R-CNN) will recognize targets and non-targets. After the targets are recognized, we employed both the k-nearest neighbors (kNN) and the fuzzy inference system (FIS) to localize two-dimension (2D) positions. Moreover, based on the field of view (FoV) and a disparity map, the depth is estimated by a mono camera mounted on the end-effector with an eye-in-hand manipulator structure. Although using a single mono camera, the system can easily find the camera baseline by only shifting the end-effector a few millimeters towards the x-axis. Thus, we can obtain and identify the depth of the layered environment in 3D points, which form a dataset to recognize the junction box covers on the table. Experimental tests confirmed that the algorithm could accurately distinguish junction box covers or non-targets and could estimate whether the targets are within the depth for grasping by three-finger grippers. Furthermore, the proposed optimized depth error of −0.0005%, and localization method could precisely position the junction box cover with recognizing and picking error rates 0.993 and 98.529% respectively.

[1]  Ian D. Reid,et al.  Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Xiao Zhang,et al.  Change Detection From Differential Airborne LiDAR Using a Weighted Anisotropic Iterative Closest Point Algorithm , 2013, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[3]  Dean Zhao,et al.  Cucumber Fruits Detection in Greenhouses Based on Instance Segmentation , 2019, IEEE Access.

[4]  Rui-Sheng Jia,et al.  Fast and Efficient Non-Contact Ball Detector for Picking Robots , 2019, IEEE Access.

[5]  Siddhartha S. Mehta,et al.  Vision-based control of robotic manipulator for citrus harvesting , 2014 .

[6]  Yu Zhang,et al.  Single Image Depth Estimation With Normal Guided Scale Invariant Deep Convolutional Fields , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[7]  QIAN ZHANG,et al.  Grasping Point Detection of Randomly Placed Fruit Cluster Using Adaptive Morphology Segmentation and Principal Component Classification of Multiple Features , 2019, IEEE Access.

[8]  Yin Zhou,et al.  VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Abdeslam Boularias,et al.  Inferring 3D Shapes of Unknown Rigid Objects in Clutter Through Inverse Physics Reasoning , 2019, IEEE Robotics and Automation Letters.

[11]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[12]  Satoshi Yamamoto,et al.  Field Operation of a Movable Strawberry-harvesting Robot using a Travel Platform , 2014 .

[13]  Lars Grimstad,et al.  Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper , 2019, Comput. Electron. Agric..

[14]  Giulio Reina,et al.  LIDAR and stereo imagery integration for safe navigation in outdoor settings , 2013, 2013 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[15]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[16]  Ahmet Burak Can,et al.  Volumetric Object Recognition Using 3-D CNNs on Depth Data , 2018, IEEE Access.

[17]  Huiyang Liao,et al.  Precision Alignment of Optical Fibers Based on Telecentric Stereo Microvision , 2016, IEEE/ASME Transactions on Mechatronics.

[18]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[19]  Ming-Shyan Wang,et al.  Eye to hand calibration using ANFIS for stereo vision-based object manipulation system , 2018 .

[20]  Jun Liang,et al.  Monocular Depth Estimation Based on Multi-Scale Graph Convolution Networks , 2020, IEEE Access.

[21]  Hyun Wook Park,et al.  Fast view interpolation of stereo images using image gradient and disparity triangulation , 2003, Signal Process. Image Commun..

[22]  Alois Knoll,et al.  Orthogonal Image Features for Visual Servoing of a 6-DOF Manipulator With Uncalibrated Stereo Cameras , 2016, IEEE Transactions on Robotics.

[23]  Quansen Sun,et al.  Joint Object Detection and Depth Estimation in Multiplexed Image , 2019, IEEE Access.

[24]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[25]  Van Anh Ho,et al.  Grasping Interface With Wet Adhesion and Patterned Morphology: Case of Thin Shell , 2019, IEEE Robotics and Automation Letters.

[26]  Yaonan Wang,et al.  Benchmark Data Set and Method for Depth Estimation From Light Field Images , 2018, IEEE Transactions on Image Processing.

[27]  Fulvio Mastrogiovanni,et al.  Detection, localisation and tracking of pallets using machine learning techniques and 2D range data , 2018, Neural Computing and Applications.

[28]  Pål Johan From,et al.  Fruit Localization and Environment Perception for Strawberry Harvesting Robots , 2019, IEEE Access.

[29]  Lin Zhang,et al.  Super-Resolution for Monocular Depth Estimation With Multi-Scale Sub-Pixel Convolutions and a Smoothness Constraint , 2019, IEEE Access.

[30]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Long Chen,et al.  Tender Tea Shoots Recognition and Positioning for Picking Robot Using Improved YOLO-V3 Model , 2019, IEEE Access.

[32]  Simon Blackmore,et al.  Development of a prototype robot and fast path-planning algorithm for static laser weeding , 2017 .

[33]  Pål Johan From,et al.  An obstacle separation method for robotic picking of fruits in clusters , 2020, Comput. Electron. Agric..

[34]  Daniela Rus,et al.  Learning Object Grasping for Soft Robot Hands , 2018, IEEE Robotics and Automation Letters.

[35]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[36]  Chi-Yi Tsai,et al.  Visual Object Recognition and Pose Estimation Based on a Deep Semantic Segmentation Network , 2018, IEEE Sensors Journal.

[37]  Jinglu Hu,et al.  Multi-Scale Dilated Convolution Network Based Depth Estimation in Intelligent Transportation Systems , 2019, IEEE Access.

[38]  Dean Zhao,et al.  Branch localization method based on the skeleton feature extraction and stereo matching for apple harvesting robot , 2017 .

[39]  Gaurav S. Sukhatme,et al.  Using Manipulation Primitives for Object Sorting in Cluttered Environments , 2015, IEEE Transactions on Automation Science and Engineering.