DEEP LEARNING FOR SEMANTIC SEGMENTATION OF 3D POINT CLOUD

Abstract. Cultural Heritage is a testimony of past human activity, and, as such, its objects exhibit great variety in their nature, size and complexity; from small artefacts and museum items to cultural landscapes, from historical building and ancient monuments to city centers and archaeological sites. Cultural Heritage around the globe suffers from wars, natural disasters and human negligence. The importance of digital documentation is well recognized and there is an increasing pressure to document our heritage both nationally and internationally. For this reason, the three-dimensional scanning and modeling of sites and artifacts of cultural heritage have remarkably increased in recent years. The semantic segmentation of point clouds is an essential step of the entire pipeline; in fact, it allows to decompose complex architectures in single elements, which are then enriched with meaningful information within Building Information Modelling software. Notwithstanding, this step is very time consuming and completely entrusted on the manual work of domain experts, far from being automatized. This work describes a method to label and cluster automatically a point cloud based on a supervised Deep Learning approach, using a state-of-the-art Neural Network called PointNet++. Despite other methods are known, we have choose PointNet++ as it reached significant results for classifying and segmenting 3D point clouds. PointNet++ has been tested and improved, by training the network with annotated point clouds coming from a real survey and to evaluate how performance changes according to the input training data. It can result of great interest for the research community dealing with the point cloud semantic segmentation, since it makes public a labelled dataset of CH elements for further tests.

[1]  Ramona Quattrini,et al.  CONSERVATION-ORIENTED HBIM. THE BIMEXPLORER WEB TOOL , 2017 .

[2]  Fabio Remondino,et al.  A REVIEW OFPOINT CLOUDS SEGMENTATION AND CLASSIFICATION ALGORITHMS , 2017 .

[3]  Anath Fischer,et al.  3D Point Cloud Classification and Segmentation using 3D Modified Fisher Vector Representation for Convolutional Neural Networks , 2017, ArXiv.

[4]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Roberto Medina,et al.  Classification of Architectural Heritage Images Using Deep Learning Techniques , 2017 .

[6]  L. Van Gool,et al.  AUTOMATIC ARCHITECTURAL STYLE RECOGNITION , 2012 .

[7]  Eduardo Zalama Casanova,et al.  Applying Deep Learning Techniques to Cultural Heritage Images Within the INCEPTION Project , 2016, EuroMed.

[8]  G. Sithole.  DETECTION OF BRICKS IN A MASONRY WALL , 2008 .

[9]  Liqiang Zhang,et al.  Large-scale urban point cloud labeling and reconstruction , 2018 .

[10]  Marc Pollefeys,et al.  Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark , 2017, ArXiv.

[11]  Facundo José López,et al.  A Framework for Using Point Cloud Data of Heritage Buildings Toward Geometry Modeling in A BIM Context: A Case Study on Santa Maria La Real De Mave Church , 2017 .

[12]  Nicolo Dell'Unto,et al.  3D GIS for Cultural Heritage Restoration : a 'white box' workflow , 2016 .

[13]  Ramona Quattrini,et al.  Knowledge-based data enrichment for HBIM: Exploring high-quality models using the semantic-web , 2017 .

[14]  Fabio Remondino,et al.  FROM 2D TO 3D SUPERVISED SEGMENTATION AND CLASSIFICATION FOR CULTURAL HERITAGE APPLICATIONS , 2018, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Barnabás Póczos,et al.  Deep Learning with Sets and Point Clouds , 2016, ICLR.

[17]  Yll Haxhimusa,et al.  Architectural Style Classification of Building Facade Windows , 2011, ISVC.

[18]  Clive S. Fraser,et al.  A TWO-STEP CLASSIFICATION APPROACH TO DISTINGUISHING SIMILAR OBJECTS IN MOBILE LIDAR POINT CLOUDS , 2017 .

[19]  S. Logothetis,et al.  Building Information Modelling for Cultural Heritage: A review , 2015 .

[20]  Emanuele Frontoni,et al.  Convolutional Networks for Semantic Heads Segmentation using Top-View Depth Data in Crowded Environment , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[21]  Xiao Liu,et al.  Recognizing architecture styles by hierarchical sparse coding of blocklets , 2014, Inf. Sci..

[22]  Fabio Remondino,et al.  3D Recording, Documentation and Management of Cultural Heritage , 2016 .

[23]  Fadi Dornaika,et al.  Image-Based Delineation and Classification of Built Heritage Masonry , 2014, Remote. Sens..

[24]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[26]  Mariarosaria Farella.  3 D MAPPING OF UNDERGROUND ENVIRONMENTS WITH A HAND-HELD LASER , 2017 .

[27]  Filip Biljecki,et al.  An improved LOD specification for 3D building models , 2016, Comput. Environ. Urban Syst..

[28]  N. Yastikli.  Documentation of cultural heritage using digital photogrammetry and laser scanning , 2007 .

[29]  Dimitri Lague,et al.  3D Terrestrial LiDAR data classification of complex natural scenes using a multi-scale dimensionality criterion: applications in geomorphology , 2011, ArXiv.

[30]  Victor S. Lempitsky,et al.  Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Fabio Remondino,et al.  Classification of 3D Digital Heritage , 2019, Remote. Sens..