Fusion of urban 3D point clouds with thermal attributes using MLS data and TIR image sequences

Abstract Buildings take a large proportion of the total energy consumption in the city area in winter, therefore Thermal Infrared (TIR) images are widely used to evaluate the energy consumption and leakage of the building. To overcome the difficulties of image interpretation and occlusion in TIR images, utilizing thermal information in 3D structures that fuse TIR images with usable georeferenced 3D coordinates of buildings, is mandatory. To accomplish this mission, we propose a strategy to generate a thermal point cloud combining Mobile Laser Scanning (MLS) point clouds and TIR images. At first, a key-points extraction method based on line-intersection is presented, which helps to detect key-points from highly distorted TIR images. Then, the semi-automatic and the automatic correspondence determination algorithm based on restricted RANSAC for 6DOF pose estimation is proposed and tested. Finally, a non-local mean strategy for data fusion is applied, which integrates the thermal attributes from image sequences and the geometric property from the point clouds, resulting in a smooth representation of thermal point clouds with comprehensive radiance property for thermal analysis. The fusion result qualitatively presents thermal radiation of elements of buildings with detailed geometric information. The generated thermal point clouds contribute to evaluating the energy efficiency of buildings, or building blocks, analyzing the invisible building structure or pipelines for retrofit, and monitoring the energy usage of buildings for long-term development.

[1]  Jianya Gong,et al.  Line-Based Registration of Panoramic Images and LiDAR Point Clouds for Mobile Mapping , 2017, Sensors.

[2]  Fakhri Karray,et al.  Multisensor data fusion: A review of the state-of-the-art , 2013, Inf. Fusion.

[3]  T. Rakha,et al.  Review of Unmanned Aerial System (UAS) applications in the built environment: Towards automated building inspection procedures using drones , 2018, Automation in Construction.

[4]  Constantinos A. Balaras,et al.  Infrared thermography for building diagnostics , 2002 .

[5]  Florian Bugarin,et al.  Automated thermal 3D reconstruction based on a robot equipped with uncalibrated infrared stereovision cameras , 2018, Adv. Eng. Informatics.

[6]  A. Adán,et al.  Fusion of thermal imagery and LiDAR data for generating TBIM models , 2017, 2017 IEEE SENSORS.

[7]  François Lauze,et al.  The Improbability of Harris Interest Points , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Ming-Der Yang,et al.  Fusion of Infrared Thermal Image and Visible Image for 3D Thermal Model Reconstruction Using Smartphone Sensors , 2018, Sensors.

[9]  Hyojoo Son,et al.  High-quality as-is 3D thermal modeling in MEP systems using a deep convolutional network , 2019, Adv. Eng. Informatics.

[10]  Olatz Irulegi,et al.  Energy Performance of Buildings , 2016 .

[11]  Uwe Stilla,et al.  Texture mapping of 3D building models with oblique direct geo-referenced airborne IRimage sequences , 2009 .

[12]  Jonathan A. Wright,et al.  Optimization of building thermal design and control by multi-criterion genetic algorithm , 2002 .

[13]  Dong Lin,et al.  Fusion of thermal imagery with point clouds for building façade thermal attribute mapping , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[14]  Peter Christiansen,et al.  Automated Detection and Recognition of Wildlife Using Thermal Cameras , 2014, Sensors.

[15]  John W. Fisher,et al.  Automatic registration of LIDAR and optical images of urban scenes , 2009, CVPR.

[16]  Federico Castanedo,et al.  A Review of Data Fusion Techniques , 2013, TheScientificWorldJournal.

[17]  F. Crosilla,et al.  PHOTOGRAMMETRIC 3D BUILDING RECONSTRUCTION FROM THERMAL IMAGES , 2017 .

[18]  Elisabetta Rosina,et al.  Mapping Infrared Data on Terrestrial Laser Scanning 3D Models of Buildings , 2011, Remote. Sens..

[19]  Hideya Takahashi,et al.  Fusion of Infrared and Visible Images for Robust Person Detection , 2011 .

[20]  T. Luhmann,et al.  GEOMETRIC CALIBRATION OF THERMOGRAPHIC CAMERAS , 2013 .

[21]  Yong K. Cho,et al.  As-Is 3D Thermal Modeling for Existing Building Envelopes Using a Hybrid LIDAR System , 2013 .

[22]  Nuno M.M. Ramos,et al.  Building information modeling for energy retrofitting – A review , 2018, Renewable and Sustainable Energy Reviews.

[23]  Konstantinos G. Derpanis,et al.  Overview of the RANSAC Algorithm , 2005 .

[24]  Torsten Sattler,et al.  Fast image-based localization using direct 2D-to-3D matching , 2011, 2011 International Conference on Computer Vision.

[25]  J. Balado,et al.  Thermal-based analysis for the automatic detection and characterization of thermal bridges in buildings , 2018 .

[26]  Sergey Korniyenko.  Evaluation of Thermal Performance of Residential Building Envelope , 2015 .

[27]  Uwe Stilla,et al.  Camera pose refinement by matching uncertain 3D building models with thermal infrared image sequences for high quality texture extraction , 2017 .

[28]  J. L. Lerma,et al.  Multitemporal thermal analysis to detect moisture on a building façade , 2011 .

[29]  James Llinas,et al.  Handbook of Multisensor Data Fusion : Theory and Practice, Second Edition , 2008 .

[30]  Rafael Grompone von Gioi,et al.  LSD: A Fast Line Segment Detector with a False Detection Control , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Uwe Stilla,et al.  THERMAL 3D MAPPING FOR OBJECT DETECTION IN DYNAMIC SCENES , 2014 .

[32]  Pedro Arias,et al.  AUTOMATIC PROCEDURE FOR THE REGISTRATION OF THERMOGRAPHIC IMAGES WITH POINT CLOUDS , 2012 .

[33]  C. Fraser.  AUTOMATIC CAMERA CALIBRATION IN CLOSE-RANGE PHOTOGRAMMETRY , 2013 .

[34]  George Vosselman,et al.  Recognizing basic structures from mobile laser scanning data for road inventory studies , 2011 .

[35]  Hideo Saito,et al.  Registration of RGB and Thermal Point Clouds Generated by Structure From Motion , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[36]  Mani Golparvar-Fard,et al.  Mapping actual thermal properties to building elements in gbXML-based BIM for reliable building energy performance modeling , 2015 .

[37]  Uwe Stilla,et al.  3D building reconstruction and construction site monitoring from RGB and TIR image sets , 2016, 2016 12th IEEE International Symposium on Electronics and Telecommunications (ISETC).

[38]  Youngjib Ham,et al.  3D as-is building energy modeling and diagnostics: A review of the state-of-the-art , 2015, Adv. Eng. Informatics.

[39]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[40]  Bisheng Yang,et al.  Hierarchical extraction of urban objects from mobile laser scanning data , 2015 .

[41]  Uwe Stilla,et al.  BUILDING FACADE OBJECT DETECTION FROM TERRESTRIAL THERMAL INFRARED IMAGE SEQUENCES COMBINING DIFFERENT VIEWS , 2015 .

[42]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[43]  Osamu Matoba,et al.  Speckle denoising by variant nonlocal means methods. , 2019, Applied optics.

[44]  Hideo Saito,et al.  Superimposing Thermal-Infrared Data on 3D Structure Reconstructed by RGB Visual Odometry , 2018, IEICE Trans. Inf. Syst..

[45]  Anna Mazur,et al.  Infrared building inspection with unmanned aerial vehicles , 2015 .

[46]  Yusheng Xu,et al.  TUM-MLS-2016: An Annotated Mobile LiDAR Dataset of the TUM City Campus for Semantic Point Cloud Interpretation in Urban Areas , 2020, Remote. Sens..

[47]  Benjamin Bustos,et al.  Harris 3D: a robust extension of the Harris operator for interest point detection on 3D meshes , 2011, The Visual Computer.

[48]  A. Habib,et al.  Photogrammetric and Lidar Data Registration Using Linear Features , 2005 .

[49]  Michael Bosse,et al.  3D thermal mapping of building interiors using an RGB-D and thermal camera , 2013, 2013 IEEE International Conference on Robotics and Automation.

[50]  Andreas Nüchter,et al.  Mutual Calibration for 3D Thermal Mapping , 2012, SyRoCo.

[51]  Youngjib Ham,et al.  Real-Time Energy Audit of Built Environments: Simultaneous Localization and Thermal Mapping , 2018, Journal of Infrastructure Systems.

[52]  Ioannis Stamos,et al.  A systematic approach for 2D-image to 3D-range registration in urban environments , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[53]  Xiao Xiang Zhu,et al.  Fusing Meter-Resolution 4-D InSAR Point Clouds and Optical Images for Semantic Urban Infrastructure Monitoring , 2017, IEEE Transactions on Geoscience and Remote Sensing.