Geometric Processing for Image-based 3D Object Modeling

Image-based 3D object modeling refers to the process of converting raw optical images to 3D digital representations of the objects. Very often, such models are desired to be dimensionally true, semantically labeled with photorealistic appearance (reality-based modeling). Laser scanning was deemed as the standard (and direct) way to obtaining highly accurate 3D measurements of objects, while one would have to abide the high acquisition cost and its unavailability on some of the platforms. Nowadays the image-based methods backboned by the recently developed advanced dense image matching algorithms and geo-referencing paradigms, are becoming the dominant approaches, due to its high flexibility, availability and low cost. The largely automated geometric processing of images in a 3D object reconstruction workflow, from ordered/unordered raw imagery to textured meshes, is becoming a critical part of the reality-based 3D modeling. This article summarizes the overall geometric processing workflow, with focuses on introducing the state-of-the-art methods of three major components of geometric processing: 1) geo-referencing; 2) Image dense matching 3) texture mapping. Finally, we will draw conclusions and share our outlooks of the topics discussed in this article.

[1]  Luiz Velho,et al.  Projective texture atlas construction for 3D photography , 2007, The Visual Computer.

[2]  Daniel Cohen-Or,et al.  Bilateral mesh denoising , 2003 .

[3]  Joost van de Weijer,et al.  Accurate Stereo Matching by Two-Step Energy Minimization , 2015, IEEE Transactions on Image Processing.

[4]  Armin Gruen NEXT GENERATION SMART CITIES-THE ROLE OF GEOMATICS , 2013 .

[5]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[6]  M. Pollefeys,et al.  Dense Semantic 3 D Reconstruction , 2016 .

[7]  J. L. Mundy The relationship between photogrammetry and computer vision , 1993 .

[8]  Rongjun Qin,et al.  An Object-Based Hierarchical Method for Change Detection Using Unmanned Aerial Vehicle Images , 2014, Remote. Sens..

[9]  Yann LeCun,et al.  Computing the stereo matching cost with a convolutional neural network , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Fabio Rocca,et al.  Permanent scatterers in SAR interferometry , 2001, IEEE Trans. Geosci. Remote. Sens..

[11]  Jean-Michel Morel,et al.  ASIFT: A New Framework for Fully Affine Invariant Image Comparison , 2009, SIAM J. Imaging Sci..

[12]  Bruno Lévy,et al.  Least squares conformal maps for automatic texture atlas generation , 2002, ACM Trans. Graph..

[13]  Anne Verroust-Blondet,et al.  Interactive texture mapping , 1993, SIGGRAPH.

[14]  Richard Bamler,et al.  Very High Resolution Spaceborne SAR Tomography in Urban Environment , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[15]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[16]  Wolfgang Förstner,et al.  Detecting interpretable and accurate scale-invariant keypoints , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[17]  A new texture mapping algorithm for photorealistic reconstruction of 3 D objects , 2008 .

[18]  Wen Gao,et al.  Local Stereo Matching with Improved Matching Cost and Disparity Refinement , 2014, IEEE MultiMedia.

[19]  Tamir Hazan,et al.  Continuous Markov Random Fields for Robust Stereo Estimation , 2012, ECCV.

[20]  Margrit Gelautz,et al.  A layered stereo matching algorithm using image segmentation and global visibility constraints , 2005 .

[21]  Horst A. Beyer,et al.  System Calibration Through Self-Calibration , 2001 .

[22]  Qingxiong Yang,et al.  Stereo Matching Using Tree Filtering , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Ruigang Yang,et al.  Spatial-Depth Super Resolution for Range Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Luc Van Gool,et al.  3D Urban Scene Modeling Integrating Recognition and Reconstruction , 2008, International Journal of Computer Vision.

[25]  T. Hanusch Texture mapping and true orthophoto generation of 3D objects , 2010 .

[26]  Ruigang Yang,et al.  Global stereo matching leveraged by sparse ground control points , 2011, CVPR 2011.

[27]  Michael S. Floater,et al.  Mean value coordinates , 2003, Comput. Aided Geom. Des..

[28]  Jan-Michael Frahm,et al.  Building Rome on a Cloudless Day , 2010, ECCV.

[29]  M. Pierrot Deseilligny,et al.  APERO, AN OPEN SOURCE BUNDLE ADJUSMENT SOFTWARE FOR AUTOMATIC CALIBRATION AND ORIENTATION OF SET OF IMAGES , 2012 .

[30]  育久 満上,et al.  Bundler: Structure from Motion for Unordered Image Collections , 2011 .

[31]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[32]  Miguel Brito,et al.  Modelling solar potential in the urban environment: State-of-the-art review , 2015 .

[33]  Pierre Boulanger,et al.  Radiometric invariant stereo matching based on relative gradients , 2012, 2012 19th IEEE International Conference on Image Processing.

[34]  Jingchao Li,et al.  3D RECONSTRUCTION BASED ON STEREOVISION AND TEXTURE MAPPING , 2010 .

[35]  Maoteng Zheng,et al.  LiDAR Strip Adjustment Using Multifeatures Matched With Aerial Images , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[36]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[37]  Lu Liu,et al.  Global Depth Refinement Based on Patches , 2017, MLICOM.

[38]  A. Habib,et al.  Bundle Adjustment with Self–Calibration Using Straight Lines , 2002 .

[39]  Marc Pollefeys,et al.  A Minimal Case Solution to the Calibrated Relative Pose Problem for the Case of Two Known Orientation Angles , 2010, ECCV.

[40]  Raquel Urtasun,et al.  Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation , 2014, ECCV.

[41]  Peng Ma Adaptive Markov random field model for dense matching of deep space stereo images , 2014 .

[42]  Takeshi Naemura,et al.  Graph Cut Based Continuous Stereo Matching Using Locally Shared Labels , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  David Nistér,et al.  Scalable Recognition with a Vocabulary Tree , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[44]  Nicolas Papadakis,et al.  Multi-label Depth Estimation for Graph Cuts Stereo Problems , 2010, Journal of Mathematical Imaging and Vision.

[45]  Roberto Cipolla,et al.  Multi-view stereo via volumetric graph-cuts , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[46]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[47]  Tony DeRose,et al.  Multiresolution analysis of arbitrary meshes , 1995, SIGGRAPH.

[48]  Qian Zhang,et al.  Classification of Ultra-High Resolution Orthophotos Combined with DSM Using a Dual Morphological Top Hat Profile , 2015, Remote. Sens..

[49]  Reinhard Klein,et al.  An Adaptable Surface Parameterization Method , 2003, IMR.

[50]  N. Haala,et al.  An update on automatic 3D building reconstruction , 2010 .

[51]  Ruigang Yang,et al.  Search Space Reduction for MRF Stereo , 2008, ECCV.

[52]  Armin Gruen,et al.  Perspectives in the reality-based generation, nD modelling, and operation of buildings and building stocks , 2009 .

[53]  Mark Meyer,et al.  Intrinsic Parameterizations of Surface Meshes , 2002, Comput. Graph. Forum.

[54]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[55]  Jean-Philippe Pons,et al.  High Accuracy and Visibility-Consistent Dense Multiview Stereo , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Xianfeng Huang,et al.  UAV project - Building a reality-based 3D model of the NUS (National University of Singapore) campus , 2012 .

[57]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[58]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  A. Gruen Algorithmic aspects in on-line triangulation , 1985 .

[60]  Michael S. Floater,et al.  Parametrization and smooth approximation of surface triangulations , 1997, Comput. Aided Geom. Des..

[61]  R. Sukthankar,et al.  PCA-SIFT: a more distinctive representation for local image descriptors , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[62]  F. Nex,et al.  UAV for 3D mapping applications: a review , 2014 .

[63]  D. Nistér,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Rudolph Triebel,et al.  Vision based interpolation of 3D laser scans , 2006 .

[65]  Qing Zhu,et al.  Multiple close‐range image matching based on a self‐adaptive triangle constraint , 2010 .

[66]  Christof Lutteroth,et al.  High-Definition Texture Reconstruction for 3D Image-based Modelling , 2013, WSCG.

[67]  Carl Olsson,et al.  In Defense of 3D-Label Stereo , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[68]  Ian F. C. Smith,et al.  A model-based data-interpretation framework for improving wind predictions around buildings , 2015 .

[69]  Zhaoqi Wang,et al.  3D entity-based stereo matching with ground control points and joint second-order smoothness prior , 2014, The Visual Computer.

[70]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[71]  Reinhard Koch,et al.  Multi Viewpoint Stereo from Uncalibrated Video Sequences , 1998, ECCV.

[72]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[73]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[74]  William T. Freeman,et al.  A Data-Driven Regularization Model for Stereo and Flow , 2014, 2014 2nd International Conference on 3D Vision.

[75]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[76]  Sunghee Choi,et al.  The power crust , 2001, SMA '01.

[77]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[78]  Naokazu Yokoya,et al.  Dense 3-D Reconstruction of an Outdoor Scene by Hundreds-Baseline Stereo Using a Hand-Held Video Camera , 2004, International Journal of Computer Vision.

[79]  M. Downey,et al.  SEMI-GLOBAL MATCHING : AN ALTERNATIVE TO LIDAR FOR DSM GENERATION ? , 2010 .

[80]  J. P. Lewis Fast Normalized Cross-Correlation , 2010 .

[81]  Fabio Remondino DETECTORS AND DESCRIPTORS FOR PHOTOGRAMMETRIC APPLICATIONS , 2006 .

[82]  Wolfgang Niem,et al.  MAPPING TEXTURE FROM MULTIPLE CAMERA VIEWS ONTO 3D-OBJECT MODELS FOR COMPUTER ANIMATION , 1995 .

[83]  S. Zlatanova,et al.  Applications of 3 D City Models : State of the Art Review , 2015 .

[84]  Heiko Hirschmüller,et al.  Evaluation of Stereo Matching Costs on Images with Radiometric Differences , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[85]  Fabio Remondino,et al.  State of the art in high density image matching , 2014 .

[86]  Guillaume Damiand,et al.  Topological Reconstruction of Complex 3D Buildings and Automatic Extraction of Levels of Detail , 2014, UDMV.

[87]  Jan-Michael Frahm,et al.  Modeling and Recognition of Landmark Image Collections Using Iconic Scene Graphs , 2008, International Journal of Computer Vision.

[88]  J. Ponce,et al.  Accurate, Dense, and Robust Multi-View Stereopsis , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[89]  Volker Coors,et al.  3D City modeling for urban scale heating energy demand forecasting , 2011 .

[90]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[91]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[92]  Yongjun Zhang,et al.  IMAGE-GUIDED NON-LOCAL DENSE MATCHING WITH THREE-STEPS OPTIMIZATION , 2016 .

[93]  Paul A. Viola,et al.  Alignment by Maximization of Mutual Information , 1997, International Journal of Computer Vision.

[94]  Pushmeet Kohli,et al.  Object stereo — Joint stereo matching and object segmentation , 2011, CVPR 2011.

[95]  Li Zhang,et al.  PMSC: PatchMatch-Based Superpixel Cut for Accurate Stereo Matching , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[96]  G. Sohn,et al.  Using a Binary Space Partitioning Tree for Reconstructing Polyhedral Building Models from Airborne Lidar Data , 2008 .

[97]  Armin Gruen,et al.  CC-MODELER : A TOPOLOGY GENERATOR FOR 3-D CITY MODELS , 1998 .

[98]  Xianfeng Huang Building reconstruction from airborne laser scanning data , 2013, Geo spatial Inf. Sci..

[99]  Xianfeng Huang,et al.  JOINT PROCESSING OF UAV IMAGERY AND TERRESTRIAL MOBILE MAPPING SYSTEM DATA FOR VERY HIGH RESOLUTION CITY MODELING , 2013 .

[100]  W. F. Clocksin,et al.  Joint Optimization for Object Class Segmentation and Dense Stereo Reconstruction , 2011, International Journal of Computer Vision.

[101]  Jae Wook Jeon,et al.  Domain Transformation-Based Efficient Cost Aggregation for Local Stereo Matching , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[102]  Richard Szeliski,et al.  Towards Internet-scale multi-view stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[103]  Sebastian Thrun,et al.  An Application of Markov Random Fields to Range Sensing , 2005, NIPS.

[104]  Frank P. Ferrie,et al.  Upsampling method for sparse light detection and ranging using coregistered panoramic images , 2015 .

[105]  Richard Szeliski,et al.  Building Rome in a day , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[106]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[107]  Victor S. Lempitsky,et al.  Seamless Mosaicing of Image-Based Texture Maps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[108]  Jan-Michael Frahm,et al.  Real-Time Visibility-Based Fusion of Depth Maps , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[109]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[110]  Petros Daras,et al.  Enhanced disparity estimation in stereo images , 2015, Image Vis. Comput..

[111]  Armin Gruen Reality-based generation of virtual environments for digital earth , 2008, Int. J. Digit. Earth.

[112]  Li Zhang Automatic Digital Surface Model (DSM) generation from linear array images , 2005 .

[113]  Andreas Geiger,et al.  Displets: Resolving stereo ambiguities using object knowledge , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[114]  K. Hormann,et al.  MIPS: An Efficient Global Parametrization Method , 2000 .

[115]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[116]  Hongyang Chao,et al.  MeshStereo: A Global Stereo Model with Mesh Alignment Regularization for View Interpolation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[117]  C. Fraser,et al.  Digital camera calibration methods: Considerations and comparisons , 2006 .

[118]  Curt H. Davis,et al.  A combined fuzzy pixel-based and object-based approach for classification of high-resolution multispectral data over urban areas , 2003, IEEE Trans. Geosci. Remote. Sens..

[119]  Markus Niederöst,et al.  Detection and reconstruction of buildings for automated map updating , 2003 .

[120]  Armin Gruen,et al.  TOBAGO — a semi-automated approach for the generation of 3-D building models , 1998 .

[121]  Bruno Lévy,et al.  Mesh parameterization: theory and practice , 2007, SIGGRAPH Courses.

[122]  O. Faugeras,et al.  Variational principles, surface evolution, PDE's, level set methods and the stereo problem , 1998, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[123]  Silvio Savarese,et al.  Dense Object Reconstruction with Semantic Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[124]  Yongjun Zhang,et al.  Direct Digital Surface Model Generation by Semi-Global Vertical Line Locus Matching , 2017, Remote. Sens..

[125]  Chang-Su Kim,et al.  Adaptive smoothness constraints for efficient stereo matching using texture and edge information , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[126]  Holly E. Rushmeier,et al.  High-Quality Texture Reconstruction from Multiple Scans , 2001, IEEE Trans. Vis. Comput. Graph..

[127]  Ron Kimmel,et al.  Texture Mapping Using Surface Flattening via Multidimensional Scaling , 2002, IEEE Trans. Vis. Comput. Graph..

[128]  In-So Kweon,et al.  Adaptive Support-Weight Approach for Correspondence Search , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[129]  Brent Schwarz,et al.  LIDAR: Mapping the world in 3D , 2010 .

[130]  Paolo Cignoni,et al.  Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models , 2008, Comput. Graph..

[131]  James M. Rehg,et al.  Joint Semantic Segmentation and 3D Reconstruction from Monocular Video , 2014, ECCV.

[132]  Cordelia Schmid,et al.  Multi-view object class detection with a 3D geometric model , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[133]  S. J. Oude Elberink,et al.  A graph edit dictionary for correcting errors in roof topology graphs reconstructed from point clouds , 2014 .

[134]  Andrea Fusiello,et al.  Improving the efficiency of hierarchical structure-and-motion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[135]  A. Gruen ADAPTIVE LEAST SQUARES CORRELATION: A POWERFUL IMAGE MATCHING TECHNIQUE , 1985 .

[136]  G. Maillet,et al.  3D CITY MODELS: AN OPERATIONAL APPROACH USING AERIAL IMAGES AND CADASTRAL MAPS , 2003 .

[137]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[138]  Xu Huang,et al.  GLOBAL PATCH MATCHING , 2017 .

[139]  Bo Wu,et al.  A Triangulation-based Hierarchical Image Matching Method for Wide-Baseline Images , 2011 .

[140]  Hans P. Moravec Obstacle avoidance and navigation in the real world by a seeing robot rover , 1980 .

[141]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[142]  Xing Mei,et al.  Stereo Matching with Reliable Disparity Propagation , 2011, 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission.

[143]  Soumik Ukil,et al.  Robust segment-based Stereo using Cost Aggregation , 2014, BMVC.

[144]  Konrad Schindler,et al.  VocMatch: Efficient Multiview Correspondence for Structure from Motion , 2014, ECCV.

[145]  Changchang Wu,et al.  Towards Linear-Time Incremental Structure from Motion , 2013, 2013 International Conference on 3D Vision.

[146]  Takeo Kanade,et al.  A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[147]  C. Fraser AUTOMATIC CAMERA CALIBRATION IN CLOSE-RANGE PHOTOGRAMMETRY , 2013 .