Ground and aerial meta-data integration for localization and reconstruction: A review

Abstract Localization and reconstruction are two highly related research areas. Both of them have developed rapidly in recent years. Apparently, with the help of ground and aerial meta-data integration, the performance of both localization and reconstruction can go a step further. For localization, aerial meta-data provides a global reference, by which the ground query can achieve a cumulative error free absolute localization. As for reconstruction, a complete and detailed model can be reconstructed by integrating ground and aerial meta-data. Though with many advantages, the integration itself is non-trivial. It is difficult to obtain ground-to-aerial correspondences neither in 2D manner nor in 3D manner. That is because: (1) The differences between the ground and aerial images in viewpoint, scale, illumination, etc. are notable; (2) The discrepancies between the ground and aerial point clouds in terms of point density, accuracy, noise level, etc. are very large. To deal with these problems, lots of methods have been proposed recently. In this paper, the methods of integrating ground and aerial meta-data for localization and reconstruction are reviewed respectively. Though many intermediate results with high quality have been achieved, we hope that inspired by the reviewed methods in this paper, more thorough methods and impressive results would emerge.

[1]  Hongyuan Zha,et al.  Multi-Graph Matching via Affinity Optimization with Graduated Consistency Regularization , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Keith Yu Kit Leung,et al.  Localization in urban environments by matching ground level video images with an aerial image , 2008, 2008 IEEE International Conference on Robotics and Automation.

[3]  Josiane Zerubia,et al.  Building Outline Extraction from Digital Elevation Models Using Marked Point Processes , 2007, International Journal of Computer Vision.

[4]  Thomas Brox,et al.  Global, Dense Multiscale Reconstruction for a Billion Points , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Bisheng Yang,et al.  Semiautomated Building Facade Footprint Extraction From Mobile LiDAR Point Clouds , 2013, IEEE Geoscience and Remote Sensing Letters.

[6]  Junchi Yan,et al.  Adaptive Discrete Hypergraph Matching , 2018, IEEE Transactions on Cybernetics.

[7]  Zhanyi Hu,et al.  Accurate mesh-based alignment for ground and aerial multi-view stereo models , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[8]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[9]  Daniel Huber,et al.  Vision based robot localization by ground to satellite matching in GPS-denied situations , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Renaud Marlet,et al.  Virtual Line Descriptor and Semi-Local Graph Matching Method for Reliable Feature Correspondence , 2012, BMVC.

[11]  Hiroshi Murase,et al.  Vehicle Ego-Localization by Matching In-Vehicle Camera Images to an Aerial Image , 2010, ACCV Workshops.

[12]  Scott Workman,et al.  Wide-Area Image Geolocalization with Aerial Reference Imagery , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[13]  S. Umeyama,et al.  Least-Squares Estimation of Transformation Parameters Between Two Point Patterns , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Lei Zhou,et al.  Progressive Large Scale-Invariant Image Matching in Scale Space , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[15]  Luc Van Gool,et al.  Efficient volumetric fusion of airborne and street-side data for urban reconstruction , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[16]  Minh N. Do,et al.  Bilateral Functions for Global Motion Modeling , 2014, ECCV.

[17]  Zhanyi Hu,et al.  Accurate and efficient ground-to-aerial model alignment , 2018, Pattern Recognit..

[18]  Gim Hee Lee,et al.  CVM-Net: Cross-View Matching Network for Image-Based Ground-to-Aerial Geo-Localization , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Jun Tang,et al.  A Laplacian spectral method for stereo correspondence , 2007, Pattern Recognit. Lett..

[20]  Alexander Bachmann,et al.  Visual features for vehicle localization and ego-motion estimation , 2009, 2009 IEEE Intelligent Vehicles Symposium.

[21]  Minh N. Do,et al.  RepMatch: Robust Feature Matching and Pose for Reconstructing Modern Cities , 2016, ECCV.

[22]  Hui Cheng,et al.  Geo-localization of street views with aerial image databases , 2011, ACM Multimedia.

[23]  Martial Hebert,et al.  Unmanned Ground Vehicle Navigation Using Aerial Ladar Data , 2006, Int. J. Robotics Res..

[24]  Larry S. Davis,et al.  Planar Structure Matching under Projective Uncertainty for Geolocation , 2014, ECCV.

[25]  Oliver Pink,et al.  Visual map matching and localization using a global feature map , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[26]  Venu Madhav Govindu,et al.  Robust Relative Rotation Averaging , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[28]  Junhao Xiao,et al.  Planar Segment Based Three‐dimensional Point Cloud Registration in Outdoor Environments , 2013, J. Field Robotics.

[29]  Silvio Savarese,et al.  Semantic Cross-View Matching , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[30]  Junjun Jiang,et al.  Robust Feature Matching for Remote Sensing Image Registration via Locally Linear Transforming , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[31]  Marco Fiocco,et al.  Multisensor fusion for volumetric reconstruction of large outdoor areas , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[32]  Zhanyi Hu,et al.  Ancient Chinese architecture 3D preservation by merging ground and aerial point clouds , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[33]  Daniel G. Aliaga,et al.  Automatic Extraction of Manhattan-World Building Masses from 3D Laser Range Scans , 2012, IEEE Transactions on Visualization and Computer Graphics.

[34]  Stergios I. Roumeliotis,et al.  An Efficient Algebraic Solution to the Perspective-Three-Point Problem , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[36]  Radu Horaud,et al.  Articulated shape matching using Laplacian eigenfunctions and unsupervised point registration , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Fabio Remondino,et al.  ISPRS benchmark for multi - platform photogrammetry , 2015 .

[39]  Hao Zhang,et al.  SmartBoxes for interactive urban reconstruction , 2010, SIGGRAPH 2010.

[40]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[41]  Bisheng Yang,et al.  An automated method to register airborne and terrestrial laser scanning point clouds , 2015 .

[42]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[43]  James Hays,et al.  Localizing and Orienting Street Views Using Overhead Imagery , 2016, ECCV.

[44]  Jean-Michel Morel,et al.  ASIFT: A New Framework for Fully Affine Invariant Image Comparison , 2009, SIAM J. Imaging Sci..

[45]  Fadi Dornaika,et al.  EXTRACTING B UILDING FOOTPRINTS FROM 3D POINT CLOUDS USING TERRESTRIAL LASER SCANNING AT STREET LEVEL , 2009 .

[46]  Christian Früh,et al.  3D model generation for cities using aerial photographs and ground level laser scans , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[47]  Fredrik Kahl,et al.  City-Scale Localization for Cameras with Known Vertical Direction , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  H. Kuhn.  The Hungarian method for the assignment problem , 1955 .

[49]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[50]  Ahmed M. Elgammal,et al.  Satellite image based precise robot localization on sidewalks , 2012, 2012 IEEE International Conference on Robotics and Automation.

[51]  Davide Scaramuzza,et al.  MAV urban localization from Google street view data , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[52]  Serge J. Belongie,et al.  Learning deep representations for ground-to-aerial geolocalization , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  David G. Lowe,et al.  Scalable Nearest Neighbor Algorithms for High Dimensional Data , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[54]  Martha Larson,et al.  Pairwise geometric matching for large-scale object retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Young-Im Cho,et al.  A 2.5D Map-Based Mobile Robot Localization via Cooperation of Aerial and Ground Robots , 2017, Sensors.

[56]  Gabriel Taubin,et al.  A Variable-Resolution Probabilistic Three-Dimensional Model for Change Detection , 2012, IEEE Transactions on Geoscience and Remote Sensing.

[57]  Yi Dong,et al.  Geo-localization using Volumetric Representations of Overhead Imagery , 2015, International Journal of Computer Vision.

[58]  Torsten Sattler,et al.  Large-Scale Location Recognition and the Geometric Burstiness Problem , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Christian Früh,et al.  Reconstructuring 3D City Models by Merging Ground-Based and Airborne Views , 2003, VLBV.

[60]  Roberto Cipolla,et al.  PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[61]  Edwin Olson,et al.  FLAG: Feature-based Localization between Air and Ground , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[62]  Steven M. Seitz,et al.  Accurate Geo-Registration by Ground-to-Aerial Image Matching , 2014, 2014 2nd International Conference on 3D Vision.

[63]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[64]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Zhanyi Hu,et al.  Global fusion of generalized camera model for efficient large-scale structure from motion , 2015, Science China Information Sciences.

[66]  Ahmed M. Elgammal,et al.  A framework for global vehicle localization using stereo images and satellite and road maps , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[67]  Michal Kedzierski,et al.  Terrestrial and Aerial Laser Scanning Data Integration Using Wavelet Analysis for the Purpose of 3D Building Modeling , 2014, Sensors.

[68]  Friedrich Fraundorfer,et al.  The TUM-DLR Multimodal Earth Observation Evaluation Benchmark , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[69]  Niloy J. Mitra,et al.  Non-local scan consolidation for 3D urban scenes , 2010, SIGGRAPH 2010.

[70]  Serge J. Belongie,et al.  Cross-View Image Geolocalization , 2013, CVPR.

[71]  Rama Chellappa,et al.  Fast directional chamfer matching , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[72]  Junjun Jiang,et al.  Guided Locality Preserving Feature Matching for Remote Sensing Image Registration , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[73]  Davide Scaramuzza,et al.  The Zurich urban micro aerial vehicle dataset , 2017, Int. J. Robotics Res..

[74]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[75]  Masatoshi Okutomi,et al.  Visual Place Recognition with Repetitive Structures , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  Yanxi Liu,et al.  Translation-Symmetry-Based Perceptual Grouping with Applications to Urban Scenes , 2010, ACCV.

[77]  Hartmut Surmann,et al.  3D mapping for multi hybrid robot cooperation , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[78]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[79]  Yanxi Liu,et al.  Regularity-Driven Building Facade Matching between Aerial and Street Views , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[80]  Mohammed Bennamoun,et al.  Rotational Projection Statistics for 3D Local Surface Description and Object Recognition , 2013, International Journal of Computer Vision.

[81]  Valérie Gouet-Brunet,et al.  A survey on Visual-Based Localization: On the benefit of heterogeneous data , 2018, Pattern Recognit..

[82]  Marcello Pelillo,et al.  Dominant Sets and Pairwise Clustering , 2007 .

[83]  Davide Scaramuzza,et al.  1-Point-RANSAC Structure from Motion for Vehicle-Mounted Cameras by Exploiting Non-holonomic Constraints , 2011, International Journal of Computer Vision.

[84]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[85]  Matthew Brand,et al.  Geolocalization using skylines from omni-images , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[86]  Scott Workman,et al.  On the location dependence of convolutional neural network features , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[87]  Tomás Pajdla,et al.  NetVLAD: CNN Architecture for Weakly Supervised Place Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[88]  Yasuyuki Matsushita,et al.  Robust Multiview Photometric Stereo Using Planar Mesh Parameterization , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[89]  Torsten Sattler,et al.  Efficient & Effective Prioritized Matching for Large-Scale Image-Based Localization , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[90]  Octavia I. Camps,et al.  Efficient City-Sized 3D Reconstruction from Ultra-High Resolution Aerial and Ground Video Imagery , 2011, ISVC.

[91]  Torsten Sattler,et al.  A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[92]  Davide Scaramuzza,et al.  Air-ground localization and map augmentation using monocular dense reconstruction , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[93]  Torsten Sattler,et al.  Camera Pose Voting for Large-Scale Image-Based Localization , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[94]  Zhanyi Hu,et al.  Tracks selection for robust, efficient and scalable large-scale structure from motion , 2017, Pattern Recognit..

[95]  Michael Isard,et al.  Object retrieval with large vocabularies and fast spatial matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[96]  Jiri Matas,et al.  Robust Detection of Lines Using the Progressive Probabilistic Hough Transform , 2000, Comput. Vis. Image Underst..

[97]  Richard Szeliski,et al.  Alignment of 3D point clouds to overhead images , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[98]  ARNO KNAPITSCH,et al.  Tanks and temples , 2017, ACM Trans. Graph..

[99]  Shuhan Shen,et al.  Accurate Multiple View 3D Reconstruction Using Patch-Based Stereo for Large-Scale Scenes , 2013, IEEE Transactions on Image Processing.

[100]  Masatoshi Okutomi,et al.  24/7 Place Recognition by View Synthesis , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[101]  Jean-Philippe Pons,et al.  Robust and Efficient Surface Reconstruction From Range Data , 2009, Comput. Graph. Forum.

[102]  Haroon Idrees,et al.  Large-Scale Image Geo-Localization Using Dominant Sets , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[103]  Mayank Bansal,et al.  Ultra-wide Baseline Facade Matching for Geo-localization , 2012, ECCV Workshops.

[104]  Tao Ju,et al.  Dual contouring of hermite data , 2002, ACM Trans. Graph..

[105]  Mubarak Shah,et al.  Cross-View Image Matching for Geo-Localization in Urban Environments , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[106]  Zhanyi Hu,et al.  HSfM: Hybrid Structure-from-Motion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[107]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[108]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[109]  Roland Siegwart,et al.  Tracking a depth camera: Parameter exploration for fast ICP , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[110]  Andrew E. Johnson,et al.  Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[111]  Jan-Michael Frahm,et al.  Pixelwise View Selection for Unstructured Multi-View Stereo , 2016, ECCV.

[112]  Gérard G. Medioni,et al.  Object modeling by registration of multiple range images , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[113]  Minh N. Do,et al.  CODE: Coherence Based Decision Boundaries for Feature Correspondence , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[114]  Manchun Li,et al.  Semi-Automatic Registration of Airborne and Terrestrial Laser Scanning Data Using Building Corner Matching with Boundaries as Reliability Check , 2013, Remote. Sens..

[115]  Zhiwei Zhu,et al.  Image to LIDAR matching for geotagging in urban environments , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).