Interactive dense point clouds in a game engine

Abstract With the development of 3D measurement systems, dense colored point clouds are increasingly available. However, up to now, their use in interactive applications has been restricted by the lack of support for point clouds in game engines. In addition, many of the existing applications for point clouds lack the capacity for fluent user interaction and application development. In this paper, we present the development and architecture of a game engine extension facilitating the interactive visualization of dense point clouds. The extension allows the development of game engine applications where users edit and interact with point clouds. To demonstrate the capabilities of the developed extension, a virtual reality head-mounted display is used and the rendering performance is evaluated. The result shows that the developed tools are sufficient for supporting real-time 3D visualization and interaction. Several promising use cases can be envisioned, including both the use of point clouds as 3D assets in interactive applications and leveraging the game engine point clouds in geomatics.

[1]  Matthias Nießner,et al.  State of the Art on 3D Reconstruction with RGB‐D Cameras , 2018, Comput. Graph. Forum.

[2]  Helmut Mayer,et al.  Incremental Division of Very Large Point Clouds for Scalable 3D Surface Reconstruction , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[3]  Luigi Barazzetti,et al.  Point cloud occlusion recovery with shallow feedforward neural networks , 2018, Adv. Eng. Informatics.

[4]  Borut Zalik,et al.  Web-Based LiDAR Visualization with Point-Based Rendering , 2011, 2011 Seventh International Conference on Signal Image Technology & Internet-Based Systems.

[5]  Yan Xu,et al.  Aerial Lidar Point Cloud Voxelization with its 3D Ground Filtering Application , 2017 .

[6]  Liu Qiyuan,et al.  Virtual reality technology applied in digitalization of cultural heritage , 2017, Cluster Computing.

[7]  Juha Hyyppä,et al.  Graph SLAM correction for single scanner MLS forest data under boreal forest canopy , 2017 .

[8]  Georgios Papaioannou,et al.  A Fast Depth-Buffer-Based Voxelization Algorithm , 1999, J. Graphics, GPU, & Game Tools.

[9]  Juha Hyyppä,et al.  Nationwide Point Cloud - The Future Topographic Core Data , 2017, ISPRS Int. J. Geo Inf..

[10]  Fabio Remondino,et al.  OBLIQUE PHOTOGRAMMETRY SUPPORTING 3D URBAN RECONSTRUCTION OF COMPLEX SCENARIOS , 2017 .

[11]  M. Bassier,et al.  Classification of sensor independent point cloud data of building objects using random forests , 2019, Journal of Building Engineering.

[12]  M. Bassier,et al.  IMMERSIVE VISUALISATION OF CONSTRUCTION SITE POINT CLOUD DATA, MESHES AND BIM MODELS IN A VR ENVIRONMENT USING A GAMING ENGINE , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[13]  Norbert Haala,et al.  An update on automatic 3D building reconstruction , 2010 .

[14]  Norbert Pfeifer,et al.  A Comprehensive Automated 3D Approach for Building Extraction, Reconstruction, and Regularization from Airborne Laser Scanning Point Clouds , 2008, Sensors.

[15]  Juha Hyyppä,et al.  Mapping Topography Changes and Elevation Accuracies Using a Mobile Laser Scanner , 2011, Remote. Sens..

[16]  Borut Zalik,et al.  Visualization of LIDAR datasets using point-based rendering technique , 2010, Comput. Geosci..

[17]  Juha Hyyppä,et al.  Assessment of Low Density Full-Waveform Airborne Laser Scanning for Individual Tree Detection and Tree Species Classification , 2014 .

[18]  Juha Hyyppä,et al.  Feasibility of Google Tango and Kinect for Crowdsourcing Forestry Information , 2017 .

[19]  Roland Billen,et al.  SMART POINT CLOUD: DEFINITION AND REMAINING CHALLENGES , 2016 .

[20]  IvanovaMilena,et al.  Massive point cloud data management , 2015 .

[21]  Elise Lachat,et al.  PERFORMANCE INVESTIGATION OF A HANDHELD 3D SCANNERTO DEFINE GOOD PRACTICES FOR SMALL ARTEFACT 3D MODELING , 2017 .

[22]  Norbert Pfeifer,et al.  Georeferenced Point Clouds: A Survey of Features and Point Cloud Management , 2013, ISPRS Int. J. Geo Inf..

[23]  Diego González-Aguilera,et al.  GRAPHOS – open‐source software for photogrammetric applications , 2018 .

[24]  Alexander Klippel,et al.  Harnessing the power of immersive virtual reality - visualization and analysis of 3D earth science data sets , 2019, Geo spatial Inf. Sci..

[25]  Sara Reardon.  A digital ark, come fire or flood , 2012 .

[26]  M. Vastaranta,et al.  Prediction of plot-level forest variables using TerraSAR-X stereo SAR data , 2012 .

[27]  Costas Armenakis,et al.  Automatic co-registration of 3D multi-sensor point clouds , 2017 .

[28]  Andreas Nüchter,et al.  One billion points in the cloud – an octree for efficient processing of 3D laser scans , 2013 .

[29]  Susanne Bleisch,et al.  Rich point clouds in virtual globes - A new paradigm in city modeling? , 2010, Comput. Environ. Urban Syst..

[30]  Kevin Ponto,et al.  Experiencing interior environments: New approaches for the immersive display of large-scale point cloud data , 2015, 2015 IEEE Virtual Reality (VR).

[31]  Joachim Hertzberg,et al.  6D SLAM—3D mapping outdoor environments , 2007, J. Field Robotics.

[32]  Renato Pajarola,et al.  Point-based rendering techniques , 2004, Comput. Graph..

[33]  Leif Kobbelt,et al.  A survey of point-based techniques in computer graphics , 2004, Comput. Graph..

[34]  Wolfram Burgard,et al.  OctoMap: an efficient probabilistic 3D mapping framework based on octrees , 2013, Autonomous Robots.

[35]  Margarita Amor,et al.  Supporting multi-resolution out-of-core rendering of massive LiDAR point clouds through non-redundant data structures , 2019, Int. J. Geogr. Inf. Sci..

[36]  J. L. Lerma,et al.  Terrestrial laser scanning and close range photogrammetry for 3D archaeological documentation: the Upper Palaeolithic Cave of Parpalló as a case study , 2010 .

[37]  Yi Lin,et al.  A low-cost multi-sensoral mobile mapping system and its feasibility for tree measurements , 2010 .

[38]  Mehdi Seyedmahmoudian,et al.  Investigating the Role of Virtual Reality in Planning for Sustainable Smart Cities , 2017 .

[39]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[40]  Julien Perret,et al.  A scalable and multi-purpose point cloud server (PCS) for easier and faster point cloud data management and processing , 2017 .

[41]  Jürgen Döllner,et al.  Out-of-Core Visualization of Classified 3D Point Clouds , 2015 .

[42]  Hannu Hyyppä,et al.  A Simple Semantic-Based Data Storage Layout for Querying Point Clouds , 2020, ISPRS Int. J. Geo Inf..

[43]  Peter van Oosterom,et al.  Massive point cloud data management: Design, implementation and execution of a point cloud benchmark , 2015, Comput. Graph..

[44]  Dieter Fox,et al.  RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments , 2012, Int. J. Robotics Res..

[45]  Roland Billen,et al.  Voxel-based 3D Point Cloud Semantic Segmentation: Unsupervised Geometric and Relationship Featuring vs Deep Learning Methods , 2019, ISPRS Int. J. Geo Inf..

[46]  Denis Laurendeau,et al.  Depth Texture Synthesis for Realistic Architectural Modeling , 2016, 2016 13th Conference on Computer and Robot Vision (CRV).

[47]  Wenzhong Shi,et al.  Semantic Geometric Modelling of Unstructured Indoor Point Cloud , 2018, ISPRS Int. J. Geo Inf..

[48]  Renato Pajarola,et al.  An efficient multi-resolution framework for high quality interactive rendering of massive point clouds using multi-way kd-trees , 2012, The Visual Computer.

[49]  Oliver Koehler,et al.  POINT CLOUD VISUALIZATION IN AN OPEN SOURCE 3D GLOBE , 2012 .

[50]  F. Poux,et al.  CLASSIFICATION AND INTEGRATION OF MASSIVE 3D POINTS CLOUDS IN A VIRTUAL REALITY (VR) ENVIRONMENT , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[51]  Juha Hyyppä,et al.  Assessing Biodiversity in Boreal Forests with UAV-Based Photogrammetric Point Clouds and Hyperspectral Imaging , 2018, Remote. Sens..

[52]  Minh-Triet Tran,et al.  Applying Virtual Reality in City Planning , 2016, HCI.

[53]  Michael Wimmer,et al.  Auto Splats: Dynamic Point Cloud Visualization on the GPU , 2012, EGPGV@Eurographics.

[54]  Marc Levoy,et al.  QSplat: a multiresolution point rendering system for large meshes , 2000, SIGGRAPH.

[55]  Hannu Hyyppä,et al.  AUTOMATED IMAGE-BASED RECONSTRUCTION OF BUILDING INTERIORS - A CASE STUDY , 2014 .

[56]  Schutz Markus,et al.  Real-Time Continuous Level of Detail Rendering of Point Clouds , 2019 .

[57]  Yangzi Cong,et al.  3D Forest Mapping Using A Low-Cost UAV Laser Scanning System: Investigation and Comparison , 2019, Remote. Sens..

[58]  Natan Micheletti,et al.  Investigating the geomorphological potential of freely available and accessible structure‐from‐motion photogrammetry using a smartphone , 2015 .

[59]  Ulrich Neumann,et al.  Visually-complete aerial LiDAR point cloud rendering , 2012, SIGSPATIAL/GIS.

[60]  Helena Rua,et al.  Living the past: 3D models, virtual reality and game engines as tools for supporting archaeology and the reconstruction of cultural heritage – the case-study of the Roman villa of Casal de Freiria , 2011 .

[61]  Sisi Zlatanova,et al.  FIRST EXPERIMENTS WITH THE TANGO TABLET FOR INDOOR SCANNING , 2016 .

[62]  Martin Kraus,et al.  Efficient image reconstruction for point-based and line-based rendering , 2008, Comput. Graph..

[63]  Juha Hyyppä,et al.  Object-based analysis of multispectral airborne laser scanner data for land cover classification and map updating , 2017 .

[64]  Zeng Xun.  WebGL based LiDAR point clouds visualization , 2012 .

[65]  Michael Wimmer,et al.  Out-of-core selection and editing of huge point clouds , 2011, Comput. Graph..

[66]  Gregory A. Gahm,et al.  Virtual reality exposure therapy for active duty soldiers. , 2008, Journal of clinical psychology.

[67]  Stephan Nebiker,et al.  Cloud-Based Geospatial 3D Image Spaces - A Powerful Urban Model for the Smart City , 2015, ISPRS Int. J. Geo Inf..

[68]  George Vosselman,et al.  Airborne and terrestrial laser scanning , 2011, Int. J. Digit. Earth.

[69]  Franco Tecchia,et al.  Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks , 2015, Interact. Learn. Environ..

[70]  Juha Hyyppä,et al.  Intelligent Open Data 3D Maps in a Collaborative Virtual World , 2015, ISPRS Int. J. Geo Inf..

[71]  Sebastián Ortega,et al.  Visualization of Large Point Cloud in Unity , 2019, Eurographics.

[72]  Juha Hyyppä,et al.  Automated Multi-Sensor 3D Reconstruction for the Web , 2019, ISPRS Int. J. Geo Inf..

[73]  Jonathan Li,et al.  Rapid Urban Roadside Tree Inventory Using a Mobile Laser Scanning System , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[74]  Bisheng Yang,et al.  An efficient global energy optimization approach for robust 3D plane segmentation of point clouds , 2018 .

[75]  Jürgen Döllner,et al.  Concepts and techniques for web-based visualization and processing of massive 3D point clouds with semantics , 2019, Graph. Model..

[76]  George Vosselman,et al.  Two algorithms for extracting building models from raw laser altimetry data , 1999 .