THERMAL 3D MAPPING FOR OBJECT DETECTION IN DYNAMIC SCENES

Abstract. The automatic analysis of 3D point clouds has become a crucial task in photogrammetry, remote sensing and computer vision. Whereas modern range cameras simultaneously provide both range and intensity images with high frame rates, other devices can be used to obtain further information which could be quite valuable for tasks such as object detection or scene interpretation. In particular thermal information offers many advantages, since people can easily be detected as heat sources in typical indoor or outdoor environments and, furthermore, a variety of concealed objects such as heating pipes as well as structural properties such as defects in isolation may be observed. In this paper, we focus on thermal 3D mapping which allows to observe the evolution of a dynamic 3D scene over time. We present a fully automatic methodology consisting of four successive steps: (i) a radiometric correction, (ii) a geometric calibration, (iii) a robust approach for detecting reliable feature correspondences and (iv) a co-registration of 3D point cloud data and thermal information via a RANSAC-based EPnP scheme. For an indoor scene, we demonstrate that our methodology outperforms other recent approaches in terms of both accuracy and applicability. We additionally show that efficient straightforward techniques allow a categorization according to background, people, passive scene manipulation and active scene manipulation.

[1]  Andreas Birk,et al.  Detecting Humans in 2D Thermal Images by Generating 3D Models , 2007, KI.

[2]  Francesca Uccheddu,et al.  Automated multispectral texture mapping of 3D models , 2009, 2009 17th European Signal Processing Conference.

[3]  Boris Jutzi.  Extending the range measurement capabilities of modulated range imaging devices by time-frequency-multiplexing , 2012 .

[4]  Boris Jutzi,et al.  Investigations on ambiguity unwrapping of range images , 2009 .

[5]  Uwe Stilla,et al.  FUSING PASSIVE AND ACTIVE SENSED IMAGES TO GAIN INFRARED-TEXTURED 3D MODELS , 2012 .

[6]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[7]  Markus Ulrich,et al.  Hierarchical real-time recognition of compound objects in images , 2003 .

[8]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[9]  Elisabetta Rosina,et al.  Mapping Infrared Data on Terrestrial Laser Scanning 3D Models of Buildings , 2011, Remote. Sens..

[10]  Zheng Liu,et al.  Context enhancement through infrared vision: a modified fusion scheme , 2007, Signal Image Video Process..

[11]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using orthonormal matrices , 1988 .

[12]  Carsten Steger,et al.  Similarity Measures for Occlusion, Clutter, and Illumination Invariant Object Recognition , 2001, DAGM-Symposium.

[13]  Surya Prakash,et al.  3D Mapping of Surface Temperature Using Thermal Stereo , 2006, 2006 9th International Conference on Control, Automation, Robotics and Vision.

[14]  Henry Leung,et al.  An EM-CI based approach to fusion of IR and visual images , 2009, 2009 12th International Conference on Information Fusion.

[15]  Boris Jutzi,et al.  A STEP TOWARDS DYNAMIC SCENE ANALYSIS WITH ACTIVE MULTI-VIEW RANGE IMAGING SYSTEMS , 2012 .

[16]  Xiangzhi Bai,et al.  Fusion of infrared and visual images through region extraction by using multi scale center-surround top-hat transform. , 2011, Optics express.

[17]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[18]  Martin Weinmann,et al.  Visual Features - From Early Concepts to Modern Computer Vision , 2013, Advanced Topics in Computer Vision.

[19]  Duane C. Brown,et al.  Close-Range Camera Calibration , 1971 .

[20]  Uwe Stilla,et al.  Detection of Windows in IR Building Textures Using Masked Correlation , 2011, PIA.

[21]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[22]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[23]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[24]  Linda G. Shapiro,et al.  Acquisition and visualization of colored 3D objects , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[25]  Beril Sirmacek,et al.  Automatic registration of Iphone images to LASER point clouds of the urban structures using shape features , 2013 .

[26]  T. Luhmann,et al.  GEOMETRIC CALIBRATION OF THERMOGRAPHIC CAMERAS , 2013 .

[27]  David G. Lowe,et al.  Fitting Parameterized Three-Dimensional Models to Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Rick S. Blum,et al.  Fusion of visual and IR images for concealed weapon detection , 2002, Proceedings of the Fifth International Conference on Information Fusion. FUSION 2002. (IEEE Cat.No.02EX5997).

[29]  Vincent Lepetit,et al.  Accurate Non-Iterative O(n) Solution to the PnP Problem , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[30]  Ioannis Stamos,et al.  A systematic approach for 2D-image to 3D-range registration in urban environments , 2012, Comput. Vis. Image Underst..

[31]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[32]  Uwe Stilla,et al.  Fusion von Time-of-Flight-Entfernungsdaten und thermalen IR-Bildern , 2014 .

[33]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[34]  Constantinos A. Balaras,et al.  Infrared thermography for building diagnostics , 2002 .

[35]  S. Umeyama,et al.  Least-Squares Estimation of Transformation Parameters Between Two Point Patterns , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Ioannis Stamos,et al.  Automatic 3D to 2D registration for the photorealistic rendering of urban scenes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[37]  Pedro Arias,et al.  Calibration and verification of thermographic cameras for geometric measurements , 2011 .

[38]  Andreas Nüchter,et al.  Thermal 3D Mapping of Building Façades , 2012, IAS.

[39]  Michael Bosse,et al.  3D thermal mapping of building interiors using an RGB-D and thermal camera , 2013, 2013 IEEE International Conference on Robotics and Automation.

[40]  Fenghui Yao,et al.  Multi-source Airborne IR and Optical Image Fusion and Its Application to Target Detection , 2008, ISVC.

[41]  Clive S. Fraser,et al.  Automatic registration of optical imagery with 3d lidar data using local combined mutual information , 2013 .

[42]  Charles K. Toth,et al.  Matching between Different Image Domains , 2011, PIA.

[43]  Andreas Nüchter,et al.  Mutual Calibration for 3D Thermal Mapping , 2012, SyRoCo.

[44]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  J. Armesto,et al.  Energy efficiency studies through 3D laser scanning and thermographic technologies , 2011 .

[46]  Severo Ochoa,et al.  A Segment-based Registration Technique for Visual-IR Images , 2000 .

[47]  Boris Jutzi,et al.  FULLY AUTOMATIC IMAGE-BASED REGISTRATION OF UNORGANIZED TLS DATA , 2012 .

[48]  Beril Sirmacek,et al.  Automatic registration of iPhone images to laser point clouds of urban structures using shape features , 2013 .

[49]  Kyung-Hoon Bae,et al.  Image fusion in infrared image and visual image using normalized mutual information , 2008, SPIE Defense + Commercial Sensing.

[50]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[51]  Uwe Stilla,et al.  Strategies for texturing Building Models with low Resolution infrared Image Sequences , 2007 .

[52]  Uwe Stilla,et al.  Photogrammetric Image Analysis , 2011, Lecture Notes in Computer Science.