Extracting Objects for Aerial Manipulation on UAVs Using Low Cost Stereo Sensors

Giving unmanned aerial vehicles (UAVs) the possibility to manipulate objects vastly extends the range of possible applications. This applies to rotary wing UAVs in particular, where their capability of hovering enables a suitable position for in-flight manipulation. Their manipulation skills must be suitable for primarily natural, partially known environments, where UAVs mostly operate. We have developed an on-board object extraction method that calculates information necessary for autonomous grasping of objects, without the need to provide the model of the object’s shape. A local map of the work-zone is generated using depth information, where object candidates are extracted by detecting areas different to our floor model. Their image projections are then evaluated using support vector machine (SVM) classification to recognize specific objects or reject bad candidates. Our method builds a sparse cloud representation of each object and calculates the object’s centroid and the dominant axis. This information is then passed to a grasping module. Our method works under the assumption that objects are static and not clustered, have visual features and the floor shape of the work-zone area is known. We used low cost cameras for creating depth information that cause noisy point clouds, but our method has proved robust enough to process this data and return accurate results.

[1]  Andrew Fitzgibbon,et al.  Gaussian Process Implicit Surfaces , 2006 .

[2]  A. Ollero,et al.  Aerial manipulation robot composed of an autonomous helicopter and a 7 degrees of freedom industrial manipulator , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[3]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[4]  Pouria Sadeghi-Tehran,et al.  A real-time approach for autonomous detection and tracking of moving objects from UAV , 2014, 2014 IEEE Symposium on Evolving and Autonomous Learning Systems (EALS).

[5]  Matthias Nießner,et al.  Combining Inertial Navigation and ICP for Real-time 3D Surface Reconstruction , 2014, Eurographics.

[6]  Changhong Fu,et al.  Efficient visual odometry and mapping for Unmanned Aerial Vehicle using ARM-based stereo vision pre-processing system , 2015, 2015 International Conference on Unmanned Aircraft Systems (ICUAS).

[7]  Karl Tuyls,et al.  OctoSLAM: A 3D mapping approach to situational awareness of unmanned aerial vehicles , 2013, 2013 International Conference on Unmanned Aircraft Systems (ICUAS).

[8]  Nico Blodow,et al.  Towards 3D Point cloud based object maps for household environments , 2008, Robotics Auton. Syst..

[9]  Andrea Fusiello,et al.  Patch-Based Background Initialization in Heavily Cluttered Video , 2010, IEEE Transactions on Image Processing.

[10]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[11]  Felipe Espinosa,et al.  Adaptive UAV Attitude Estimation Employing Unscented Kalman Filter, FOAM and Low-Cost MEMS Sensors , 2012, Sensors.

[12]  Yasir Mohd Mustafah,et al.  Colour-based Object Detection and Tracking for Autonomous Quadrotor UAV , 2013 .

[13]  Marc Toussaint,et al.  Gaussian process implicit surfaces for shape estimation and grasping , 2011, 2011 IEEE International Conference on Robotics and Automation.

[14]  Radu Bogdan Rusu,et al.  Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments , 2010, KI - Künstliche Intelligenz.

[15]  Simon Lacroix,et al.  Vision-Based SLAM: Stereo and Monocular Approaches , 2007, International Journal of Computer Vision.

[16]  Aníbal Ollero,et al.  Control of an aerial robot with multi-link arm for assembly tasks , 2013, 2013 IEEE International Conference on Robotics and Automation.

[17]  Alexander Ferrein,et al.  IR Stereo Kinect: Improving Depth Images by Combining Structured Light with IR Stereo , 2014, PRICAI.

[18]  Jorge Dias,et al.  Relative Pose Calibration Between Visual and Inertial Sensors , 2007, Int. J. Robotics Res..

[19]  Hyung Jin Chang,et al.  Detection of Moving Objects with Non-stationary Cameras in 5.8ms: Bringing Motion Detection to Your Mobile Device , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[21]  Oyas Wahyunggoro,et al.  UAV obstacle avoidance using potential field under dynamic environment , 2015, 2015 International Conference on Control, Electronics, Renewable Energy and Communications (ICCEREC).

[22]  Aníbal Ollero,et al.  Localization and mapping for aerial manipulation based on range-only measurements and visual markers , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Ales Ude,et al.  Pushing and grasping for autonomous learning of object models with foveated vision , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[24]  Konstantin Kondak,et al.  Closed-Loop Behavior of an Autonomous Helicopter Equipped with a Robotic Arm for Aerial Manipulation Tasks , 2013 .

[25]  Ramesh C. Jain,et al.  Building an environment model using depth information , 1989, Computer.

[26]  Aníbal Ollero,et al.  Control of a multirotor outdoor aerial manipulator , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Sauro Longhi,et al.  A Biased Extended Kalman Filter for Indoor Localization of a Mobile Agent Using Low-Cost IMU and UWB Wireless Sensor Network , 2012, SyRoCo.

[28]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Yusuf Sinan Akgül,et al.  3D reconstruction with mirrors and RGB-D cameras , 2015, 2014 International Conference on Computer Vision Theory and Applications (VISAPP).

[30]  Allen R. Hanson,et al.  Coherent Motion Segmentation in Moving Camera Videos Using Optical Flow Orientations , 2013, 2013 IEEE International Conference on Computer Vision.

[31]  Marc Van Droogenbroeck,et al.  ViBe: A Universal Background Subtraction Algorithm for Video Sequences , 2011, IEEE Transactions on Image Processing.

[32]  Aníbal Ollero,et al.  Lightweight compliant arm for aerial manipulation , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[33]  Milton C. P. Santos,et al.  UAV obstacle avoidance using RGB-D system , 2015, 2015 International Conference on Unmanned Aircraft Systems (ICUAS).

[34]  Shih-Chia Huang,et al.  An Advanced Motion Detection Algorithm With Video Quality Analysis for Video Surveillance Systems , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[35]  Tamim Asfour,et al.  Discovery, segmentation and reactive grasping of unknown objects , 2012, 2012 12th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2012).

[36]  J. Braga,et al.  Aerial manipulator for structure inspection by contact from the underside , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[37]  Gonzalo Pajares,et al.  Overview and Current Status of Remote Sensing Applications Based on Unmanned Aerial Vehicles (UAVs) , 2015 .

[38]  Witold Czajewski,et al.  Moving Objects Detection and Tracking Framework for UAV-based Surveillance , 2010, 2010 Fourth Pacific-Rim Symposium on Image and Video Technology.

[39]  Stefan Hrabar,et al.  3D path planning and stereo-based obstacle avoidance for rotorcraft UAVs , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[40]  Jörg Stückler,et al.  Large-scale direct SLAM with stereo cameras , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[41]  Stefan Winkler,et al.  A no-reference perceptual blur metric , 2002, Proceedings. International Conference on Image Processing.

[42]  Shih-Chia Huang,et al.  A Hybrid Background Subtraction Method with Background and Foreground Candidates Detection , 2015, ACM Trans. Intell. Syst. Technol..

[43]  Bruce A. MacDonald,et al.  A Real-Time Method to Detect and Track Moving Objects (DATMO) from Unmanned Aerial Vehicles (UAVs) Using a Single Camera , 2012, Remote. Sens..

[44]  Aníbal Ollero,et al.  A multilayer control for multirotor UAVs equipped with a servo robot arm , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[45]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[46]  H. S. Wolff,et al.  iRun: Horizontal and Vertical Shape of a Region-Based Graph Compression , 2022, Sensors.

[47]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[48]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[49]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[50]  Anton Satria Prabuwono,et al.  Motion analysis for moving object detection from UAV aerial images: A review , 2014, 2014 International Conference on Informatics, Electronics & Vision (ICIEV).

[51]  Isabelle Herlin,et al.  Computer Vision – ECCV 2012 , 2012, Lecture Notes in Computer Science.

[52]  Andrew Price,et al.  Real time object detection for an unmanned aerial vehicle using an FPGA based vision system , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[53]  Shih-Chia Huang,et al.  Probabilistic neural networks based moving vehicles extraction algorithm for intelligent traffic surveillance systems , 2015, Inf. Sci..

[54]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.