UAV Pose Estimation in GNSS-Denied Environment Assisted by Satellite Imagery Deep Learning Features

With the growing maturity of unmanned aerial vehicle (UAV) technology, its applications have widened to many spheres of life. The prerequisite for a UAV to perform air tasks smoothly is an accurate localization of its own position. Traditional UAV navigation relies on the Global Navigation Satellite System (GNSS) for localization; however, this system has disadvantages of instability and susceptibility to interference. Therefore, to obtain accuracy in UAV pose estimation in GNSS-denied environments, a UAV localization method that is assisted by deep learning features of satellite imagery is proposed. With the inclusion of a top-view optical camera to the UAV, localization is achieved based on satellite imageries with geographic coordinates and a digital elevation model (DEM). By utilizing the difference between the UAV frame and satellite imagery, the convolutional neural network is used to extract deep learning features between the two images to achieve stable registration. To improve the accuracy and robustness of the localization method, a local optimization method based on bundle adjustment (BA) is proposed. Experiments demonstrate that when the UAV’s relative altitude is 0.5 km, the average localization error of this method under different trajectories is within 15 m.

[1]  Torsten Sattler,et al.  D2-Net: A Trainable CNN for Joint Description and Detection of Local Features , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Rokas Jurevičius,et al.  Robust GNSS-denied localization for UAV using particle filter and visual odometry , 2019, Machine Vision and Applications.

[3]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[4]  Wilfried Linder,et al.  Digital Photogrammetry , 2003 .

[5]  Pratap Tokekar,et al.  Sensor planning for a symbiotic UAV and UGV system for precision agriculture , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[7]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[8]  Simon Lucey,et al.  GPS-Denied UAV Localization using Pre-existing Satellite Imagery , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[9]  Davide Scaramuzza,et al.  SVO: Fast semi-direct monocular visual odometry , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  G. Conte,et al.  An Integrated UAV Navigation System Based on Aerial Image Matching , 2008, 2008 IEEE Aerospace Conference.

[12]  Grace Xingxin Gao,et al.  UAV Pose Estimation using Cross-view Geolocalization with Satellite Imagery , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[13]  Mariana Luderitz Kolberg,et al.  A novel measurement model based on abBRIEF for global localization of a UAV over satellite images , 2019, Robotics Auton. Syst..

[14]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[15]  Friedrich Fraundorfer,et al.  Visual Odometry Part I: The First 30 Years and Fundamentals , 2022 .

[16]  Yang Liu,et al.  Visual loop closure detection with a compact image descriptor , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Ethan Rublee,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[18]  Jochen Teizer,et al.  Mobile 3D mapping for surveying earthwork projects using an Unmanned Aerial Vehicle (UAV) system , 2014 .

[19]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[20]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[21]  J. Carroll.  Vulnerability Assessment of the U.S. Transportation Infrastructure that Relies on the Global Positioning System , 2003 .

[22]  Gordon Wyeth,et al.  OpenFABMAP: An open source toolbox for appearance-based loop closure detection , 2012, 2012 IEEE International Conference on Robotics and Automation.

[23]  Éric Marchand,et al.  Second-Order Optimization of Mutual Information for Real-Time Image Registration , 2012, IEEE Transactions on Image Processing.

[24]  Bohyung Han,et al.  Large-Scale Image Retrieval with Attentive Deep Local Features , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Xiaolei Yang,et al.  Research on Feature Extraction of Tumor Image Based on Convolutional Neural Network , 2019, IEEE Access.

[26]  Bernhard Rinner,et al.  An Autonomous Multi-UAV System for Search and Rescue , 2015, DroNet@MobiSys.

[27]  Yongwei Sheng,et al.  Theoretical Analysis of the Iterative Photogrammetric Method to Determining Ground Coordinates from Photo Coordinates and a DEM , 2005 .

[28]  Daniel Huber,et al.  Vision-based robot localization across seasons and in remote locations , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[29]  P. Wong,et al.  A UAV-Mounted Whole Cell Biosensor System for Environmental Monitoring Applications , 2015, IEEE Transactions on NanoBioscience.

[30]  Aníbal Ollero,et al.  Improving vision-based planar motion estimation for unmanned aerial vehicles through online mosaicing , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[31]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[32]  Ahmed M. Elgammal,et al.  Satellite image based precise robot localization on sidewalks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[33]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Fei Wang,et al.  Google map aided visual navigation for UAVs in GPS-denied environment , 2015, 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[35]  Reda ElHakim,et al.  A Deep CNN-Based Framework For Enhanced Aerial Imagery Registration with Applications to UAV Geolocalization , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[36]  Gaurav S. Sukhatme,et al.  Sliding window filter with application to planetary landing , 2010, J. Field Robotics.

[37]  Andrew J. Davison,et al.  A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).