How to Build a 2D and 3D Aerial Multispectral Map? - All Steps Deeply Explained

The increased development of camera resolution, processing power, and aerial platforms helped to create more cost-efficient approaches to capture and generate point clouds to assist in scientific fields. The continuous development of methods to produce three-dimensional models based on two-dimensional images such as Structure from Motion (SfM) and Multi-View Stereopsis (MVS) allowed to improve the resolution of the produced models by a significant amount. By taking inspiration from the free and accessible workflow made available by OpenDroneMap, a detailed analysis of the processes is displayed in this paper. As of the writing of this paper, no literature was found that described in detail the necessary steps and processes that would allow the creation of digital models in two or three dimensions based on aerial images. With this, and based on the workflow of OpenDroneMap, a detailed study was performed. The digital model reconstruction process takes the initial aerial images obtained from the field survey and passes them through a series of stages. From each stage, a product is acquired and used for the following stage, for example, at the end of the initial stage a sparse reconstruction is produced, obtained by extracting features of the images and matching them, which is used in the following step, to increase its resolution. Additionally, from the analysis of the workflow, adaptations were made to the standard workflow in order to increase the compatibility of the developed system to different types of image sets. Particularly, adaptations focused on thermal imagery were made. Due to the low presence of strong features and therefore difficulty to match features across thermal images, a modification was implemented, so thermal models could be produced alongside the already implemented processes for multispectral and RGB image sets.

[1]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[2]  Tao Wang,et al.  Fast stitching of DOM based on small UAV , 2017 .

[3]  José Barata,et al.  Semantic Navigation Mapping from Aerial Multispectral Imagery , 2019, 2019 IEEE 28th International Symposium on Industrial Electronics (ISIE).

[4]  Holly E. Rushmeier,et al.  High-Quality Texture Reconstruction from Multiple Scans , 2001, IEEE Trans. Vis. Comput. Graph..

[5]  Szymon Rusinkiewicz,et al.  Poisson Surface Reconstruction with Envelope Constraints , 2020, Comput. Graph. Forum.

[6]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[7]  Jie Liao,et al.  Texture Mapping for 3D Reconstruction with RGB-D Sensor , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[9]  F. Frontera,et al.  PRELIMINARY INVESTIGATION INTO THE GEOMETRIC CALIBRATION OF THE MICASENSE REDEDGE-M MULTISPECTRAL CAMERA , 2020 .

[10]  GoeseleMichael,et al.  Fusion of depth maps with multiple scales , 2011 .

[11]  Gabriela Csurka,et al.  Visual categorization with bags of keypoints , 2002, eccv 2004.

[12]  Richard Szeliski,et al.  Modeling surfaces of arbitrary topology with dynamic particles , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Alex M. Andrew Object Recognition By Computer: The Role Of Geometric Constraints by W. Eric, L. Grimson, with contributions from Tomás Lazano-Pérez and Daniel P. Huttenlocher, MIT Press, Cambridge, Mass., 1990, Hard cover, xv + 512 pp. (£40.50) , 1992, Robotica.

[14]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Cordelia Schmid,et al.  An Affine Invariant Interest Point Detector , 2002, ECCV.

[16]  Simon Fuhrmann,et al.  Fusion of depth maps with multiple scales , 2011, ACM Trans. Graph..

[17]  Pierre Alliez,et al.  Noname manuscript No. (will be inserted by the editor) Feature-Preserving Surface Reconstruction and Simplification from Defect-Laden Point Sets , 2013 .

[18]  Jon Louis Bentley,et al.  An Algorithm for Finding Best Matches in Logarithmic Expected Time , 1977, TOMS.

[19]  Pascal Monasse,et al.  Multi-view Texturing of Imprecise Mesh , 2009, ACCV.

[20]  Michael Wimmer,et al.  Fast Out‐of‐Core Octree Generation for Massive Point Clouds , 2020, Comput. Graph. Forum.

[21]  Yaping Zhang,et al.  A High-Realistic Texture Mapping Algorithm Based on Image Sequences , 2018, 2018 26th International Conference on Geoinformatics.

[22]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[23]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[24]  D. Marr,et al.  Representation and recognition of the spatial organization of three-dimensional shapes , 1978, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[25]  S. J. Oude Elberink,et al.  EVALUATING THE QUALITY OF PHOTOGRAMMETRIC POINT-CLOUDS IN CHALLENGING GEO-ENVIRONMENTS – A CASE STUDY IN AN ALPINE VALLEY , 2020 .

[26]  Sunghee Choi,et al.  The power crust , 2001, SMA '01.

[27]  Marc Pollefeys,et al.  Interactive 3D architectural modeling from unordered photo collections , 2008, SIGGRAPH 2008.

[28]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[29]  Meenakshisundaram Gopi,et al.  Surface Reconstruction based on Lower Dimensional Localized Delaunay Triangulation , 2000, Comput. Graph. Forum.

[30]  Markus H. Gross,et al.  Feature Preserving Point Set Surfaces based on Non‐Linear Kernel Regression , 2009, Comput. Graph. Forum.

[31]  Shuhan Shen,et al.  Accurate Multiple View 3D Reconstruction Using Patch-Based Stereo for Large-Scale Scenes , 2013, IEEE Transactions on Image Processing.

[32]  Michael Goesele,et al.  Let There Be Color! Large-Scale Texturing of 3D Reconstructions , 2014, ECCV.

[33]  Meenakshisundaram Gopi,et al.  A fast and efficient projection-based approach for surface reconstruction , 2002, Proceedings. XV Brazilian Symposium on Computer Graphics and Image Processing.

[34]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[35]  João Costa Seco,et al.  UAV Cloud Platform for Precision Farming , 2020, 2020 12th International Symposium on Communication Systems, Networks and Digital Signal Processing (CSNDSP).

[36]  Gabriel Taubin,et al.  An improved algorithm for algebraic curve and surface fitting , 1993, 1993 (4th) International Conference on Computer Vision.

[37]  Daniel Cremers,et al.  Integration of Multiview Stereo and Silhouettes Via Convex Functionals on Convex Domains , 2008, ECCV.

[38]  Michael M. Kazhdan,et al.  An Adaptive Multi‐Grid Solver for Applications in Computer Graphics , 2019, Comput. Graph. Forum.

[39]  Andrew J. Hanson,et al.  Hyperquadrics: Smoothly deformable shapes with convex polyhedral bounds , 1988, Comput. Vis. Graph. Image Process..

[40]  U. Ascher,et al.  Consolidation of unorganized point clouds for surface reconstruction , 2009, ACM Trans. Graph..

[41]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Laura Inzerillo,et al.  HIGH QUALITY TEXTURE MAPPING PROCESS AIMED AT THE OPTIMIZATION OF 3D STRUCTURED LIGHT MODELS , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[43]  Nelly Pustelnik,et al.  Proximity Operator of a Sum of Functions; Application to Depth Map Estimation , 2017, IEEE Signal Processing Letters.

[44]  Cordelia Schmid,et al.  Local Grayvalue Invariants for Image Retrieval , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Konrad Schindler,et al.  VocMatch: Efficient Multiview Correspondence for Structure from Motion , 2014, ECCV.

[46]  Hui Zhao,et al.  Conformal mesh parameterization using discrete Calabi flow , 2018, Comput. Aided Geom. Des..

[47]  TaubinGabriel,et al.  The Ball-Pivoting Algorithm for Surface Reconstruction , 1999 .

[48]  Jan-Michael Frahm,et al.  Efficient and Scalable Depthmap Fusion , 2012, BMVC.

[49]  Dimitris N. Metaxas,et al.  Dynamic 3D Models with Local and Global Deformations: Deformable Superquadrics , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[50]  N. Jamwal,et al.  A survey on depth map estimation strategies , 2016 .

[51]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[52]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[53]  D. DeCarlo,et al.  Blended Deformable Models , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Rolf Klein,et al.  Voronoi Diagrams and Delaunay Triangulations , 2013, Encyclopedia of Algorithms.

[55]  Livio Pinto,et al.  Experimental analysis of different software packages for orientation and digital surface modelling from UAV images , 2014, Earth Science Informatics.

[56]  Uwe Stilla,et al.  Automatic 3D reconstruction and texture extraction for 3D building models from thermal infrared image sequences. , 2016 .

[57]  Randal C. Burns,et al.  Parallel Poisson Surface Reconstruction , 2009, ISVC.

[58]  Marc Pollefeys,et al.  Multi-View Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[59]  Hans Hagen,et al.  Surface Reconstruction from Unorganized 3D Point Clouds , 2012 .

[60]  David G. Lowe,et al.  Local feature view clustering for 3D object recognition , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[61]  Alok Gupta,et al.  The extruded generalized cylinder: a deformable model for object recovery , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[62]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[63]  Matthew A. Brown,et al.  Invariant Features from Interest Point Groups , 2002, BMVC.

[64]  Ruigang Yang,et al.  Fast Texture Mapping Adjustment via Local/Global Optimization , 2019, IEEE Transactions on Visualization and Computer Graphics.

[65]  Dana H. Ballard,et al.  Generalizing the Hough transform to detect arbitrary shapes , 1981, Pattern Recognit..

[66]  Hong Wei,et al.  Correction of image radial distortion based on division model , 2017 .

[67]  C. Fletcher Computational Galerkin Methods , 1983 .

[68]  Michael Goesele,et al.  Multi-View Stereo Revisited , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[69]  Ravi Ramamoorthi,et al.  Patch-based optimization for image-based texture mapping , 2017, ACM Trans. Graph..

[70]  Hui Zeng,et al.  A Novel Approach for Interest Point Detection via Laplacian-of-Bilateral Filter , 2015, J. Sensors.

[71]  Jean-Philippe Pons,et al.  Towards high-resolution large-scale multi-view stereo , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[72]  Xiao Cheng,et al.  An improved Poisson Surface Reconstruction algorithm , 2010, 2010 International Conference on Audio, Language and Image Processing.

[73]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[74]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[75]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[76]  Barr,et al.  Superquadrics and Angle-Preserving Transformations , 1981, IEEE Computer Graphics and Applications.

[77]  André Mora,et al.  FFAU - Framework for Fully Autonomous UAVs , 2020, Remote. Sens..

[78]  Luís Pádua,et al.  VisWebDrone: A Web Application for UAV Photogrammetry Based on Open-Source Software , 2020, ISPRS Int. J. Geo Inf..

[79]  Ligang Liu,et al.  Progressive parameterizations , 2018, ACM Trans. Graph..

[80]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[81]  J. Koenderink The structure of images , 2004, Biological Cybernetics.

[82]  Marshall W. Bern,et al.  A new Voronoi-based surface reconstruction algorithm , 1998, SIGGRAPH.

[83]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[84]  F. Neitzel,et al.  Mobile 3d Mapping with a Low-Cost Uav System , 2012 .

[85]  Long Quan,et al.  Cross-Atlas Convolution for Parameterization Invariant Learning on Textured Mesh Surface , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[86]  Lin Zhang,et al.  A Novel OpenMVS-Based Texture Reconstruction Method Based on the Fully Automatic Plane Segmentation for 3D Mesh Models , 2020, Remote. Sens..

[87]  Demetri Terzopoulos,et al.  Constraints on Deformable Models: Recovering 3D Shape and Nonrigid Motion , 1988, Artif. Intell..

[88]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[89]  Shigeru Muraki,et al.  Volumetric shape description of range data using “Blobby Model” , 1991, SIGGRAPH.

[90]  Nassir Navab,et al.  Coloured signed distance fields for full 3D object reconstruction , 2014, BMVC.

[91]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[92]  Long Quan,et al.  A quasi-dense approach to surface reconstruction from uncalibrated images , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Daniel Cohen-Or,et al.  Seamless Montage for Texturing Models , 2010, Comput. Graph. Forum.

[94]  Kyungdon Joo,et al.  Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation , 2021, IEEE Robotics and Automation Letters.

[95]  Paolo Cignoni,et al.  Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models , 2008, Comput. Graph..

[96]  Filiberto Chiabrando,et al.  Accuracy Assessment of a UAV Block by Different Software Packages, Processing Schemes and Validation Strategies , 2020, ISPRS Int. J. Geo Inf..

[97]  Jean-Philippe Pons,et al.  Seamless image-based texture atlases using multi-band blending , 2008, 2008 19th International Conference on Pattern Recognition.

[98]  T. Lindeberg,et al.  Scale-Space Theory : A Basic Tool for Analysing Structures at Different Scales , 1994 .

[99]  Bruce A. Draper,et al.  Introduction to the Bag of Features Paradigm for Image Classification and Retrieval , 2011, ArXiv.

[100]  Marc Alexa,et al.  Computing and Rendering Point Set Surfaces , 2003, IEEE Trans. Vis. Comput. Graph..

[101]  Kenneth Levenberg A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[102]  Georgios D. Evangelidis,et al.  Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[103]  Derek Hoiem,et al.  Improving Structure from Motion with Reliable Resectioning , 2020, 2020 International Conference on 3D Vision (3DV).

[104]  Ulrich Reif,et al.  Weighted Extended B-Spline Approximation of Dirichlet Problems , 2001, SIAM J. Numer. Anal..

[105]  Michael Goesele,et al.  Multi-View Stereo for Community Photo Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[106]  John E. Hopcroft,et al.  The Geometry of Projective Blending Surfaces , 1988, Artif. Intell..

[107]  Victor S. Lempitsky,et al.  Seamless Mosaicing of Image-Based Texture Maps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[108]  Greg Turk,et al.  Reconstructing surfaces using anisotropic basis functions , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[109]  Eric Q. Li,et al.  Bundled depth-map merging for multi-view stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[110]  Jan-Michael Frahm,et al.  Real-Time Visibility-Based Fusion of Depth Maps , 2007, 2007 IEEE 11th International Conference on Computer Vision.