Automated point cloud classification using an image-based instance segmentation for structure from motion

Abstract Point cloud constantly gains popularity as a visualization tool in numerous fields including civil infrastructure scope. However, automatic point cloud classification for civil infrastructures such as piers is challenging due to untidy scenes, gigantic sizes, and image feature-rich objects that can generate many cloud points. Moreover, the lack of training point clouds and unrealistic synthetic data preventing deep learning to fully support the three-dimensional point cloud classification. This paper proposes Point cloud Classification based on image-based Instance Segmentation (PCIS), an automated point cloud classification based on two-dimensional digital images from a daily work basis. These images are processed into the pre-trained network in PCIS to generate mask images, which are later used to create three-dimensional masks based on the projection from the solved camera parameters. The cloud points located inside these masks are classified as the cloud point of interest. The experiment result showed that PCIS correctly classified the point cloud and achieved up to 0.96 F1-score from a one-class classification sample and 0.83 F1-score from a six-class classification sample in our validation process. Our study has proved that normal digital images can also be used to train deep learning to classify the three-dimensional point cloud.

[1]  Yong Jae Lee,et al.  YOLACT: Real-Time Instance Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[2]  Joel Nothman,et al.  SciPy 1.0-Fundamental Algorithms for Scientific Computing in Python , 2019, ArXiv.

[3]  Di Wang,et al.  Unsupervised semantic and instance segmentation of forest point clouds , 2020 .

[4]  Ekaterina Merkurjev,et al.  A fast graph-based data classification method with applications to 3D sensory data in the form of point clouds , 2020, Pattern Recognit. Lett..

[5]  Stephen Gareth Pierce,et al.  Novel algorithms for 3D surface point cloud boundary detection and edge reconstruction , 2019, J. Comput. Des. Eng..

[6]  Abbas Rashidi,et al.  Video to BrIM: Automated 3D As-Built Documentation of Bridges , 2018 .

[7]  Qingwu Hu,et al.  A POINT CLOUD CLASSIFICATION APPROACH BASED ON VERTICAL STRUCTURES OF GROUND OBJECTS , 2018 .

[8]  David A. Holland,et al.  Automatic 3d Building Reconstruction from a Dense Image Matching Dataset , 2016 .

[9]  J. Brasington,et al.  Object-based land cover classification using airborne LiDAR , 2008 .

[10]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[11]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[12]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[13]  S. Ullman.  The interpretation of structure from motion , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[14]  Xiang Yang,et al.  A Single Point-Based Multilevel Features Fusion and Pyramid Neighborhood Optimization Method for ALS Point Cloud Classification , 2019, Applied Sciences.

[15]  Gabriel Walton,et al.  Classification methods for point clouds in rock slope monitoring: A novel machine learning approach and comparative analysis , 2019, Engineering Geology.

[16]  Matthieu Guillaumin,et al.  Non-maximum Suppression for Object Detection by Passing Messages Between Windows , 2014, ACCV.

[17]  Wanshou Jiang,et al.  Reliable image matching via photometric and geometric constraints structured by Delaunay triangulation , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[18]  Anirban Mukhopadhyay,et al.  Habitat-Net: Segmentation of habitat images using deep learning , 2018, bioRxiv.

[19]  Hasan Asy'ari Arief,et al.  Addressing Overfitting on Pointcloud Classification using Atrous XCRF , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[20]  Mohammad Sohel Rahman,et al.  MultiResUNet : Rethinking the U-Net Architecture for Multimodal Biomedical Image Segmentation , 2019, Neural Networks.

[21]  M. H. Alipour,et al.  Visualization of As-Built Progress Data Using ConstructionSite Photographs: Two Case Studies , 2014 .

[22]  Xin Wang,et al.  Structure from motion for ordered and unordered image sets based on random k-d forests and global pose estimation , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[23]  Georgios Triantafyllidis,et al.  Real-time, anthropomorphic 3-D scanning and voxel display system using consumer depth cameras as an interactive means of individual artistic expression through light , 2018 .

[24]  Norbert Pfeifer,et al.  Classification of image matching point clouds over an urban area , 2018 .

[25]  N. Altman.  An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression , 1992 .

[26]  Xiang Li,et al.  Directionally Constrained Fully Convolutional Neural Network For Airborne Lidar Point Cloud Classification , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[27]  Juntao Yang,et al.  A probabilistic graphical model for the classification of mobile LiDAR point clouds , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[28]  Konrad Schindler,et al.  Joint classification and contour extraction of large 3D point clouds , 2017 .

[29]  Markus H. Gross,et al.  A Network Architecture for Point Cloud Classification via Automatic Depth Images Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Geoffrey J. McLachlan,et al.  Analyzing Microarray Gene Expression Data , 2004 .

[31]  Nobuyoshi Yabuki,et al.  Development of an unwanted-feature removal system for Structure from Motion of repetitive infrastructure piers using deep learning , 2020, Adv. Eng. Informatics.

[32]  Patricio A. Vela,et al.  Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future , 2015, Adv. Eng. Informatics.

[33]  Lin Gao,et al.  VV-Net: Voxel VAE Net With Group Convolutions for Point Cloud Segmentation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[34]  Jean-Michel Guldmann,et al.  Creating 3D city models with building footprints and LIDAR point cloud classification: A machine learning approach , 2019, Comput. Environ. Urban Syst..

[35]  Jiasong Wu,et al.  Instance Segmentation Network With Self-Distillation for Scene Text Detection , 2020, IEEE Access.

[36]  José García Rodríguez,et al.  PointNet: A 3D Convolutional Neural Network for real-time object class recognition , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[37]  Qing Zhu,et al.  Robust point cloud classification based on multi-level semantic relationships for urban scenes , 2017 .

[38]  Jean-Emmanuel Deschaud,et al.  Classification of Point Cloud for Road Scene Understanding with Multiscale Voxel Deep Network , 2018 .

[39]  Yong K. Cho,et al.  A Point Cloud-Vision Hybrid Approach for 3D Location Tracking of Mobile Construction Assets , 2016 .

[40]  David Moloney,et al.  1.2 Watt Classification of 3D Voxel Based Point-clouds using a CNN on a Neural Compute Stick , 2020, Neurocomputing.

[41]  Andrew Pulver,et al.  Railcar Detection, Identification and Tracking for Rail Yard Management , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[42]  Jaehoon Jung,et al.  Automated and efficient powerline extraction from laser scanning data using a voxel-based subsampling with hierarchical approach , 2020 .

[43]  Richard Szeliski,et al.  Towards Internet-scale multi-view stereo , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[44]  Ioannis Brilakis,et al.  Concrete Column Recognition in Images and Videos , 2010, J. Comput. Civ. Eng..

[45]  Lei Wang,et al.  3D Point Cloud Analysis and Classification in Large-Scale Scene Based on Deep Learning , 2019, IEEE Access.

[46]  Hao Shen,et al.  CenterMask: Single Shot Instance Segmentation With Point Representation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Pileun Kim,et al.  UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments , 2019, Automation in Construction.

[48]  José García Rodríguez,et al.  A survey on deep learning techniques for image and video semantic segmentation , 2018, Appl. Soft Comput..

[49]  Grégoire Guillet,et al.  Camera orientation, calibration and inverse perspective with uncertainties: A Bayesian method applied to area estimation from diverse photographs , 2020 .

[50]  Yue Zhang,et al.  Rotation-aware and multi-scale convolutional neural network for object detection in remote sensing images , 2020 .

[51]  Hao Chen,et al.  BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Saifullahi Aminu Bello,et al.  Review: deep learning on 3D point clouds , 2020, Remote. Sens..

[53]  Duc Thanh Nguyen,et al.  Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[54]  Jean-Philippe Pons,et al.  High Accuracy and Visibility-Consistent Dense Multiview Stereo , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Ioannis Pratikakis,et al.  Ensemble of PANORAMA-based convolutional neural networks for 3D model classification and retrieval , 2017, Comput. Graph..

[56]  M. Westoby,et al.  ‘Structure-from-Motion’ photogrammetry: A low-cost, effective tool for geoscience applications , 2012 .

[57]  Hujun Bao,et al.  Deep Snake for Real-Time Instance Segmentation , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Mauricio Galo,et al.  PHOTOGRAMMETRIC POINT CLOUD CLASSIFICATION BASED ON GEOMETRIC AND RADIOMETRIC DATA INTEGRATION , 2019 .

[59]  David Brie,et al.  Local surface sampling step estimation for extracting boundaries of planar point clouds , 2016 .

[60]  Ioannis Brilakis,et al.  Progressive 3D reconstruction of infrastructure with videogrammetry , 2011 .

[61]  Yusheng Xu,et al.  Pairwise coarse registration of point clouds in urban scenes using voxel-based 4-planes congruent sets , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[62]  H. Rastiveis,et al.  Automated extraction of lane markings from mobile LiDAR point clouds based on fuzzy inference , 2020 .

[63]  Bryan T. Adey,et al.  Use of Unmanned Aerial Vehicle Photogrammetry to Obtain Topographical Information to Improve Bridge Risk Assessment , 2018 .

[64]  Mohammed Bennamoun,et al.  NormalNet: A voxel-based CNN for 3D object classification and retrieval , 2019, Neurocomputing.

[65]  Anuj Srivastava,et al.  Detection, classification and estimation of individual shapes in 2D and 3D point clouds , 2013, Comput. Stat. Data Anal..