Grasp Planning and Visual Servoing for an Outdoors Aerial Dual Manipulator

Abstract This paper describes a system for grasping known objects with unmanned aerial vehicles (UAVs) provided with dual manipulators using an RGB-D camera. Aerial manipulation remains a very challenging task. This paper covers three principal aspects for this task: object detection and pose estimation, grasp planning, and in-flight grasp execution. First, an artificial neural network (ANN) is used to obtain clues regarding the object’s position. Next, an alignment algorithm is used to obtain the object’s six-dimensional (6D) pose, which is filtered with an extended Kalman filter. A three-dimensional (3D) model of the object is then used to estimate an arranged list of good grasps for the aerial manipulator. The results from the detection algorithm—that is, the object’s pose—are used to update the trajectories of the arms toward the object. If the target poses are not reachable due to the UAV’s oscillations, the algorithm switches to the next feasible grasp. This paper introduces the overall methodology, and provides the experimental results of both simulation and real experiments for each module, in addition to a video showing the results.

[1]  Takeo Kanade,et al.  Automated Construction of Robotic Manipulation Programs , 2010 .

[2]  Dieter Fox,et al.  PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes , 2017, Robotics: Science and Systems.

[3]  Patrick Bouthemy,et al.  Robust real-time visual tracking using a 2D-3D model-based approach , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Aníbal Ollero,et al.  A multilayer control for multirotor UAVs equipped with a servo robot arm , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Guillermo Heredia,et al.  Design of an Anthropomorphic, Compliant, and Lightweight Dual Arm for Aerial Manipulation , 2018, IEEE Access.

[7]  Konstantin Kondak,et al.  High accuracy visual servoing for aerial manipulation using a 7 degrees of freedom industrial manipulator , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  James J. Kuffner,et al.  OpenRAVE: A Planning Architecture for Autonomous Robotics , 2008 .

[9]  Faouzi Ghorbel,et al.  A simple and efficient approach for 3D mesh approximate convex decomposition , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[10]  Zijun Zhang,et al.  Automatic Detection of Wind Turbine Blade Surface Cracks Based on UAV-Taken Images , 2017, IEEE Transactions on Industrial Electronics.

[11]  Manuela M. Veloso,et al.  Fast and inexpensive color image segmentation for interactive robots , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[12]  Lakmal Seneviratne,et al.  A survey on inspecting structures using robotic systems , 2016 .

[13]  Arnold W. M. Smeulders,et al.  Color-based object recognition , 1997, Pattern Recognit..

[14]  Rs Roel Pieters,et al.  Visual Servo Control , 2012 .

[15]  Pierre-Luc Richard,et al.  LineScout Technology Opens the Way to Robotic Inspection and Maintenance of High-Voltage Power Lines , 2015, IEEE Power and Energy Technology Systems Journal.

[16]  Aníbal Ollero,et al.  Bridge Mapping for Inspection Using an UAV Assisted by a Total Station , 2017, ROBOT.

[17]  J. F. Reinoso,et al.  Cartography for Civil Engineering Projects: Photogrammetry Supported by Unmanned Aerial Vehicles , 2018 .

[18]  Ales Leonardis,et al.  Dynamic grasp and trajectory planning for moving objects , 2018, Autonomous Robots.

[19]  Hao Men,et al.  Color point cloud registration with 4D ICP algorithm , 2011, 2011 IEEE International Conference on Robotics and Automation.

[20]  Pablo Ramon Soria,et al.  Geometric Priors for Gaussian Process Implicit Surfaces , 2017, IEEE Robotics and Automation Letters.

[21]  Juha Hyyppä,et al.  Remote sensing methods for power line corridor surveys , 2016 .

[22]  Beno Benhabib,et al.  A complete generalized solution to the inverse kinematics of robots , 1985, IEEE J. Robotics Autom..

[23]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[24]  David G. Lowe,et al.  What and Where: 3D Object Recognition with Accurate Pose , 2006, Toward Category-Level Object Recognition.

[25]  Nicolas Courty,et al.  Inverse Kinematics Using Sequential Monte Carlo Methods , 2008, AMDO.

[26]  Aníbal Ollero,et al.  Detection, Location and Grasping Objects Using a Stereo Sensor on UAV in Outdoor Environments , 2017, Sensors.

[27]  S. Buss.  Introduction to Inverse Kinematics with Jacobian Transpose , Pseudoinverse and Damped Least Squares methods , 2004 .

[28]  Rohit Gupta,et al.  Grasping Region Identification in Novel Objects Using Microsoft Kinect , 2012, ICONIP.

[29]  Nancy M. Amato,et al.  Fast approximate convex decomposition using relative concavity , 2013, Comput. Aided Des..

[30]  Marc Toussaint,et al.  Gaussian process implicit surfaces for shape estimation and grasping , 2011, 2011 IEEE International Conference on Robotics and Automation.

[31]  Henrik I. Christensen,et al.  Automatic grasp planning using shape primitives , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[32]  Kaiming He,et al.  Mask R-CNN , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Gary R. Bradski,et al.  Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor , 2012, Robotics: Science and Systems.

[34]  François Chaumette,et al.  Visual servo control. I. Basic approaches , 2006, IEEE Robotics & Automation Magazine.

[35]  Huiyu Zhou,et al.  Object tracking using SIFT features and mean shift , 2009, Comput. Vis. Image Underst..

[36]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[37]  Jianwei Zhang,et al.  Learning to grasp everyday objects using reinforcement-learning with automatic value cut-off , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[38]  Matko Orsag,et al.  Towards valve turning using a dual-arm aerial manipulator , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Aníbal Ollero,et al.  Aerial Manipulation: A Literature Review , 2018, IEEE Robotics and Automation Letters.

[40]  Rüdiger Dillmann,et al.  An efficient grasp planning algorithm based on decomposition of grasp regions , 2012, 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012).

[41]  R. Amirfattahi,et al.  Real-time pose estimation and tracking of rigid objects in 3D space using extended Kalman filter , 2014, 2014 22nd Iranian Conference on Electrical Engineering (ICEE).

[42]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[43]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[44]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[45]  Danica Kragic,et al.  Enhancing visual perception of shape through tactile glances , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[46]  Peter Corke,et al.  Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach , 2018, Robotics: Science and Systems.

[47]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[48]  Danica Kragic,et al.  Learning task constraints for robot grasping using graphical models , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Máximo A. Roa,et al.  Grasp quality measures: review and performance , 2014, Autonomous Robots.

[50]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[51]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[52]  Quanquan Shao,et al.  Real time pose estimation based on extended Kalman filter for binocular camera , 2016, 2016 Asia-Pacific Conference on Intelligent Robot Systems (ACIRS).

[53]  A. Ollero,et al.  Lightweight and human-size dual arm aerial manipulator , 2017, 2017 International Conference on Unmanned Aircraft Systems (ICUAS).

[54]  Siddhartha S. Srinivasa,et al.  Object recognition and full pose registration from a single image for robotic manipulation , 2009, 2009 IEEE International Conference on Robotics and Automation.

[55]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Larry Leifer,et al.  Applications of Damped Least-Squares Methods to Resolved-Rate and Resolved-Acceleration Control of Manipulators , 1988 .

[57]  Peter I. Corke,et al.  A tutorial on visual servo control , 1996, IEEE Trans. Robotics Autom..

[58]  H. Jin Kim,et al.  Vision-Guided Aerial Manipulation Using a Multirotor With a Robotic Arm , 2016, IEEE/ASME Transactions on Mechatronics.