Efficient edge-aware surface mesh reconstruction for urban scenes

Meshing approach for both street-side SfM data and large-scale urban height maps.Our meshes preserve crease edges and discontinuities without staircasing artifacts.2D base mesh from superpixels or from piecewise-planar depth map partitioning.Fast linear vertex depth optimization including a curvature penalty term.Excellent trade-off between model compactness and approximation quality. We propose an efficient approach for building compact, edge-preserving, view-centric triangle meshes from either dense or sparse depth data, with a focus on modeling architecture in large-scale urban scenes. Our method constructs a 2D base mesh from a preliminary view partitioning, then lifts the base mesh into 3D in a fast vertex depth optimization. Different view partitioning schemes are proposed for imagery and dense depth maps. They guarantee that mesh edges are aligned with crease edges and discontinuities. In particular, we introduce an effective plane merging procedure with a global error guarantee in order to maximally compact the resulting models. Moreover, different strategies for detecting and handling discontinuities are presented. We demonstrate that our approach provides an excellent trade-off between quality and compactness, and is eligible for fast production of polyhedral building models from large-scale urban height maps, as well as, for direct meshing of sparse street-side Structure-from-Motion(SfM) data.

[1]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[3]  J. Mixter Fast , 2012 .

[4]  Luc Van Gool,et al.  A Three-Layered Approach to Facade Parsing , 2012, ECCV.

[5]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[6]  Richard Szeliski,et al.  Manhattan-world stereo , 2009, CVPR.

[7]  Horst Bischof,et al.  Incremental Surface Extraction from Sparse Structure-from-Motion Point Clouds , 2013, BMVC.

[8]  Jie Chen,et al.  Architectural Modeling from Sparsely Scanned Range Data , 2008, International Journal of Computer Vision.

[9]  Adrian Hilton,et al.  Scene modelling from sparse 3D data , 2005, Image Vis. Comput..

[10]  Marc Pollefeys,et al.  Photometric Bundle Adjustment for Dense Multi-view 3D Modeling , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Christian Früh,et al.  Data Processing Algorithms for Generating Textured 3D Building Facade Meshes from Laser Scans and Camera Images , 2005, International Journal of Computer Vision.

[12]  Ruigang Yang,et al.  Global stereo matching leveraged by sparse ground control points , 2011, CVPR 2011.

[13]  Luc Van Gool,et al.  An Integer Linear Programming Model for View Selection on Overlapping Camera Clusters , 2014, 2014 2nd International Conference on 3D Vision.

[14]  Hans-Peter Seidel,et al.  A multi-scale approach to 3D scattered data interpolation with compactly supported basis functions , 2003, 2003 Shape Modeling International..

[15]  Hayko Riemenschneider,et al.  Irregular lattices for complex shape grammar facade parsing , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Ulrich Neumann,et al.  2.5D Dual Contouring: A Robust Approach to Creating Building Models from Aerial LiDAR Point Clouds , 2010, ECCV.

[17]  Richard Szeliski,et al.  First-person hyper-lapse videos , 2014, ACM Trans. Graph..

[18]  Horst Bischof,et al.  Fusion of Feature- and Area-Based Information for Urban Buildings Modeling from Aerial Imagery , 2008, ECCV.

[19]  Zheng Zhi A Region Based Stereo Matching Algorithm Using Cooperative Optimization , 2009 .

[20]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Jean-Philippe Pons,et al.  Robust piecewise-planar 3D reconstruction and completion from large-scale unstructured point data , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[23]  Andrew Zisserman,et al.  New Techniques for Automated Architectural Reconstruction from Photographs , 2002, ECCV.

[24]  Tom Drummond,et al.  ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition , 2009, BMVC.

[25]  Konrad Schindler,et al.  Piecewise planar scene reconstruction from sparse correspondences , 2006, Image Vis. Comput..

[26]  Martin Kada,et al.  3D BUILDING RECONSTRUCTION FROM LIDAR BASED ON A CELL DECOMPOSITION APPROACH , 2009 .

[27]  Leif Kobbelt,et al.  A Surface-Growing Approach to Multi-View Stereo Reconstruction , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Jianxiong Xiao,et al.  Image-based façade modeling , 2008, SIGGRAPH 2008.

[29]  Luc Van Gool,et al.  3D all the way: Semantic segmentation of urban scenes from start to end in 3D , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ashutosh Saxena,et al.  Learning 3-D Scene Structure from a Single Still Image , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[31]  Florent Lafarge,et al.  Creating Large-Scale City Models from 3D-Point Clouds: A Robust Approach with Hybrid Representation , 2012, International Journal of Computer Vision.

[32]  Jan-Michael Frahm,et al.  Piecewise planar and non-planar stereo for urban scene reconstruction , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Timo Pylvänäinen,et al.  3D City Modeling from Street-Level Data for Augmented Reality Applications , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[34]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[35]  Jean-Philippe Pons,et al.  High Accuracy and Visibility-Consistent Dense Multiview Stereo , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Roberto Cipolla,et al.  Multi-view stereo via volumetric graph-cuts , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[37]  Michael Garland,et al.  Surface simplification using quadric error metrics , 1997, SIGGRAPH.

[38]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[39]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[40]  Luc Van Gool,et al.  Learning Where to Classify in Multi-view Semantic Segmentation , 2014, ECCV.

[41]  Josiane Zerubia,et al.  Building Outline Extraction from Digital Elevation Models Using Marked Point Processes , 2007, International Journal of Computer Vision.

[42]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[43]  Ulrich Neumann,et al.  2.5D building modeling by discovering global regularities , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Josiane Zerubia,et al.  Structural Approach for Building Reconstruction from a Single DSM , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Jan-Michael Frahm,et al.  3D Reconstruction Using an n-Layer Heightmap , 2010, DAGM-Symposium.

[46]  Hugues Hoppe,et al.  Progressive meshes , 1996, SIGGRAPH.

[47]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Long Quan,et al.  Region-based progressive stereo matching , 2004, CVPR 2004.

[49]  David H. Douglas,et al.  ALGORITHMS FOR THE REDUCTION OF THE NUMBER OF POINTS REQUIRED TO REPRESENT A DIGITIZED LINE OR ITS CARICATURE , 1973 .

[50]  Luc Van Gool,et al.  Superpixel meshes for fast edge-preserving surface reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Maxime Lhuillier A Generic Error Model and Its Application to Automatic 3D Modeling of Scenes Using a Catadioptric Camera , 2010, International Journal of Computer Vision.

[52]  Davide Scaramuzza,et al.  REMODE: Probabilistic, monocular dense reconstruction in real time , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[53]  Andrew J. Davison,et al.  Live dense reconstruction with a single moving camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[54]  Pascal Fua,et al.  On benchmarking camera calibration and multi-view stereo for high resolution imagery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  Sing Bing Kang,et al.  Stereo for Image-Based Rendering using Image Over-Segmentation , 2007, International Journal of Computer Vision.

[56]  Ulrich Neumann,et al.  2.5D building modeling with topology control , 2011, CVPR 2011.

[57]  David G. Kirkpatrick,et al.  On the shape of a set of points in the plane , 1983, IEEE Trans. Inf. Theory.

[58]  Michela Spagnuolo,et al.  Shape Analysis and Structuring (Mathematics and Visualization) , 2007 .

[59]  Luc Van Gool,et al.  3D Urban Scene Modeling Integrating Recognition and Reconstruction , 2008, International Journal of Computer Vision.

[60]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[61]  Richard Szeliski,et al.  Piecewise planar stereo for image-based rendering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[62]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[63]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[64]  Maxime Lhuillier Toward automatic 3D modeling of scenes using a generic camera model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[65]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[66]  D. Nistér,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Steven M. Seitz,et al.  Multicore bundle adjustment , 2011, CVPR 2011.

[68]  Roberto Cipolla,et al.  Bayesian Stochastic Mesh Optimization for 3D reconstruction , 2003, BMVC.

[69]  Marc Pollefeys,et al.  Pulling Things out of Perspective , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[70]  Maxime Lhuillier,et al.  Manifold surface reconstruction of an environment from sparse Structure-from-Motion data , 2013, Comput. Vis. Image Underst..

[71]  Jianxiong Xiao,et al.  Image-based façade modeling , 2008, ACM Trans. Graph..

[72]  Andrew J. Davison,et al.  DTAM: Dense tracking and mapping in real-time , 2011, 2011 International Conference on Computer Vision.

[73]  Daniel G. Aliaga,et al.  A Survey of Urban Reconstruction , 2013, Comput. Graph. Forum.

[74]  András Bódis-Szomorú,et al.  Fast, Approximate Piecewise-Planar Modeling Based on Sparse Structure-from-Motion and Superpixels , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[75]  Luc Van Gool,et al.  SEEDS: Superpixels Extracted via Energy-Driven Sampling , 2012, ECCV.

[76]  Yuichi Taguchi,et al.  Stereo reconstruction with mixed pixels using adaptive over-segmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[77]  Suya You,et al.  Automatic reconstruction of cities from remote sensor data , 2009, CVPR.

[78]  Jan-Michael Frahm,et al.  Detailed Real-Time Urban 3D Reconstruction from Video , 2007, International Journal of Computer Vision.

[79]  T. Pock,et al.  Point Clouds: Lidar versus 3D Vision , 2010 .

[80]  Jana Kosecka,et al.  Multi-view Superpixel Stereo in Urban Environments , 2010, International Journal of Computer Vision.

[81]  Pierre Alliez,et al.  Polygon Mesh Processing , 2010 .

[82]  Jean-Philippe Pons,et al.  Efficient Multi-View Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[83]  Pascal Fua,et al.  Efficient large-scale multi-view stereo for ultra high-resolution image sets , 2011, Machine Vision and Applications.

[84]  Hai Tao,et al.  A global matching framework for stereo computation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[85]  Florent Lafarge,et al.  Insertion of 3-D-Primitives in Mesh-Based Representations: Towards Compact Models Preserving the Details , 2010, IEEE Transactions on Image Processing.

[86]  Pierre Alliez,et al.  Mesh approximation using a volume-based metric , 1999, Proceedings. Seventh Pacific Conference on Computer Graphics and Applications (Cat. No.PR00293).

[87]  Horst Bischof,et al.  Image-based Building Classification and 3D Modelling with Super-Pixels , 2010 .

[88]  Mariette Yvinec,et al.  Surface Reconstruction from MultiView Stereo ∗ , 2009 .

[89]  Olivier D. Faugeras,et al.  Representing Stereo Data with the Delaunay Triangulation , 1990, Artif. Intell..

[90]  Marc Pollefeys,et al.  Live Metric 3D Reconstruction on Mobile Phones , 2013, 2013 IEEE International Conference on Computer Vision.

[91]  Changchang Wu,et al.  SiftGPU : A GPU Implementation of Scale Invariant Feature Transform (SIFT) , 2007 .

[92]  Jarek Rossignac,et al.  Multi-resolution 3D approximations for rendering complex scenes , 1993, Modeling in Computer Graphics.

[93]  Julius Ziegler,et al.  StereoScan: Dense 3d reconstruction in real-time , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[94]  Pascal Fua,et al.  A parallel stereo algorithm that produces dense depth maps and preserves image features , 1993, Machine Vision and Applications.

[95]  Mathieu Desbrun,et al.  Variational shape approximation , 2004, SIGGRAPH 2004.

[96]  Norbert Haala,et al.  An update on automatic 3D building reconstruction , 2010 .

[97]  Florent Lafarge,et al.  LOD Generation for Urban Scenes , 2015, ACM Trans. Graph..

[98]  Renaud Marlet,et al.  A MRF shape prior for facade parsing with occlusions , 2015, CVPR.

[99]  Camillo J. Taylor Surface reconstruction from feature based stereo , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[100]  Takeo Kanade,et al.  Image-consistent surface triangulation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[101]  Roberto Cipolla,et al.  Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[102]  Florent Lafarge,et al.  Structure‐Aware Mesh Decimation , 2015, Comput. Graph. Forum.

[103]  Daniel G. Aliaga,et al.  Automatic urban modeling using volumetric reconstruction with surface graph cuts , 2013, Comput. Graph..

[104]  Luc Van Gool,et al.  Image-based procedural modeling of facades , 2007, SIGGRAPH 2007.

[105]  Ashutosh Saxena,et al.  3-D Reconstruction from Sparse Views using Monocular Vision , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[106]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..