BigSUR

The creation of high-quality semantically parsed 3D models for dense metropolitan areas is a fundamental urban modeling problem. Although recent advances in acquisition techniques and processing algorithms have resulted in large-scale imagery or 3D polygonal reconstructions, such data-sources are typically noisy, and incomplete, with no semantic structure. In this paper, we present an automatic data fusion technique that produces high-quality structured models of city blocks. From coarse polygonal meshes, street-level imagery, and GIS footprints, we formulate a binary integer program that globally balances sources of error to produce semantically parsed mass models with associated facade elements. We demonstrate our system on four city regions of varying complexity; our examples typically contain densely built urban blocks spanning hundreds of buildings. In our largest example, we produce a structured model of 37 city blocks spanning a total of 1, 011 buildings at a scale and quality previously impossible to achieve automatically.

[1]  Shimin Hu,et al.  Adaptive Partitioning of Urban Facades , 2011 .

[2]  Dong-Ming Yan,et al.  Automatic Constraint Detection for 2D Layout Regularization , 2016, IEEE Transactions on Visualization and Computer Graphics.

[3]  John W. Chinneck,et al.  Feasibility and Infeasibility in Optimization:: Algorithms and Computational Methods , 2007 .

[4]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Bernard Ghanem,et al.  Large Scale Asset Extraction for Urban Images , 2016, ECCV.

[6]  N. Mitra,et al.  Non-local scan consolidation for 3D urban scenes , 2010, ACM Trans. Graph..

[7]  Daniel G. Aliaga,et al.  Building reconstruction using manhattan-world grammars , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Roberto Cipolla,et al.  Segmentation and Recognition Using Structure from Motion Point Clouds , 2008, ECCV.

[9]  Roberto Cipolla,et al.  Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding , 2015, BMVC.

[10]  C. Brenner Building reconstruction from images and laser scanning , 2005 .

[11]  Niloy J. Mitra,et al.  RAPter , 2015, ACM Trans. Graph..

[12]  Florent Lafarge,et al.  Structure‐Aware Mesh Decimation , 2015, Comput. Graph. Forum.

[13]  Bernard Ghanem,et al.  Template Assembly for Detailed Urban Reconstruction , 2015, Comput. Graph. Forum.

[14]  Roberto Cipolla,et al.  Modelling and Interpretation of Architecture from Several Images , 2004, International Journal of Computer Vision.

[15]  Luc Van Gool,et al.  Learning Domain Knowledge for Façade Labelling , 2012, ECCV.

[16]  Peter Wonka,et al.  Manhattan-World Urban Reconstruction from Point Clouds , 2016, ECCV.

[17]  Ulrich Neumann,et al.  2.5D Dual Contouring: A Robust Approach to Creating Building Models from Aerial LiDAR Point Clouds , 2010, ECCV.

[18]  D. Cohen-Or,et al.  SmartBoxes for interactive urban reconstruction , 2010, ACM Trans. Graph..

[19]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Franz Aurenhammer,et al.  A Novel Type of Skeleton for Polygons , 1995, J. Univers. Comput. Sci..

[21]  Radim Tylecek,et al.  The CMP Facade Database , 2012 .

[22]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[23]  Peter Palfrader,et al.  Planar Matchings for Weighted Straight Skeletons , 2014, ISAAC.

[24]  Daniel Cohen-Or,et al.  GlobFit: consistently fitting primitives by discovering global relations , 2011, ACM Trans. Graph..

[25]  Hayko Riemenschneider,et al.  Irregular lattices for complex shape grammar facade parsing , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Daniel Cohen-Or,et al.  2D-3D fusion for layer decomposition of urban facades , 2011, 2011 International Conference on Computer Vision.

[27]  Luc Van Gool,et al.  A Three-Layered Approach to Facade Parsing , 2012, ECCV.

[28]  Florent Lafarge,et al.  LOD Generation for Urban Scenes , 2015, ACM Trans. Graph..

[29]  Renaud Marlet,et al.  A MRF shape prior for facade parsing with occlusions , 2015, CVPR.

[30]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[31]  A. Paoluzzi,et al.  MODELING AND RENDERING , 2010 .

[32]  David Eppstein,et al.  Raising roofs, crashing cycles, and playing pool: applications of a data structure for finding pairwise interactions , 1998, SCG '98.

[33]  Daniel G. Aliaga,et al.  A Survey of Urban Reconstruction , 2013, Comput. Graph. Forum.

[34]  Marc Pollefeys,et al.  Efficient Structured Parsing of Facades Using Dynamic Programming , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Daniel G. Aliaga,et al.  Interactive sketching of urban procedural models , 2016, ACM Trans. Graph..

[36]  Andrew M. Day,et al.  Automatically generating large urban environments based on the footprint data of buildings , 2003, SM '03.

[37]  Supun Samarasekera,et al.  Building segmentation for densely built urban regions using aerial LIDAR data , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Suya You,et al.  Automatic reconstruction of cities from remote sensor data , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[40]  Niloy J. Mitra,et al.  Discovering Structured Variations Via Template Matching , 2017, Comput. Graph. Forum.

[41]  Luc Van Gool,et al.  Image-based procedural modeling of facades , 2007, ACM Trans. Graph..

[42]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[43]  Peter Palfrader,et al.  Weighted straight skeletons in the plane☆ , 2013, CCCG.

[44]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.

[45]  Luc Van Gool,et al.  Procedural modeling of buildings , 2006, SIGGRAPH 2006.

[46]  Iasonas Kokkinos,et al.  Parsing Facades with Shape Grammars and Reinforcement Learning , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Peter Wonka,et al.  Interactive architectural modeling with procedural extrusions , 2011, TOGS.

[48]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[49]  Daniel G. Aliaga,et al.  Automatic urban modeling using volumetric reconstruction with surface graph cuts , 2013, Comput. Graph..

[50]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[51]  Sebastian Ramos,et al.  The Cityscapes Dataset for Semantic Urban Scene Understanding , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Franz Aurenhammer,et al.  A Novel Type of Skeleton for Polygons , 1996 .

[53]  Jianxiong Xiao,et al.  Image-based façade modeling , 2008, ACM Trans. Graph..

[54]  Vladimir G. Kim,et al.  Shape-based recognition of 3D point clouds in urban environments , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[55]  Josiane Zerubia,et al.  Structural Approach for Building Reconstruction from a Single DSM , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Long Quan,et al.  Image-Based Modeling of Unwrappable Façades , 2013, IEEE Transactions on Visualization and Computer Graphics.

[57]  Christian Früh,et al.  Google Street View: Capturing the World at Street Level , 2010, Computer.

[58]  Chao Yang,et al.  Parsing façade with rank-one approximation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Niloy J. Mitra,et al.  Coupled structure-from-motion and 3D symmetry detection for urban facades , 2014, ACM Trans. Graph..

[60]  David Eppstein,et al.  Raising Roofs, Crashing Cycles, and Playing Pool: Applications of a Data Structure for Finding Pairwise Interactions , 1998, SCG '98.

[61]  Hui Lin,et al.  Semantic decomposition and reconstruction of residential scenes from LiDAR data , 2013, ACM Trans. Graph..