Multimodal Dense Stereo Matching

In this paper, we propose a new approach for dense depth estimation based on multimodal stereo images. Our approach employs a combined cost function utilizing robust metrics and a transformation to an illumination independent representation. Additionally, we present a confidence based weighting scheme which allows a pixel-wise weight adjustment within the cost function. We demonstrate the capabilities of our approach using RGB- and thermal images. The resulting depth maps are evaluated by comparing them to depth measurements of a Velodyne HDL-64E LiDAR sensor. We show that our method outperforms current state of the art dense matching methods regarding depth estimation based on multimodal input images.

[1]  Bir Bhanu,et al.  Fusion of color and infrared video for moving human detection , 2007, Pattern Recognit..

[2]  Sebastian P. Kleinschmidt,et al.  Spatial Fusion of Different Imaging Technologies Using a Virtual Multimodal Camera , 2016, ICINCO.

[3]  Peter Kovesi,et al.  Phase Congruency Detects Corners and Edges , 2003, DICTA.

[4]  H. Jones,et al.  Combining thermal and visible imagery for estimating canopy temperature and identifying plant stress. , 2004, Journal of experimental botany.

[5]  Sebastian P. Kleinschmidt,et al.  Visual Multimodal Odometry: Robust Visual Odometry in Harsh Environments , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[6]  C. Heipke,et al.  Illumination Invariant Dense Image Matching based on Sparse Features , 2018 .

[7]  Peter Kovesi,et al.  Image Features from Phase Congruency , 1995 .

[8]  J.P. Heather,et al.  Multimodal image registration with applications to image fusion , 2005, 2005 7th International Conference on Information Fusion.

[9]  Robert Pless,et al.  Extrinsic Auto-calibration of a Camera and Laser Range Finder , 2003 .

[10]  Seungryong Kim,et al.  Mahalanobis Distance Cross-Correlation for Illumination-Invariant Stereo Matching , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[11]  Xiaoyan Hu,et al.  A Quantitative Evaluation of Confidence Measures for Stereo Vision , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  S. Ribaric,et al.  Thermal and Visual Image Registration in Hough Parameter Space , 2007, 2007 14th International Workshop on Systems, Signals and Image Processing and 6th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services.

[13]  Peyman Moghadam,et al.  HeatWave : a handheld 3D thermography system for energy auditing , 2013 .

[14]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[15]  Belur V. Dasarathy,et al.  Medical Image Fusion: A survey of the state of the art , 2013, Inf. Fusion.

[16]  C. Heipke,et al.  Multi-view dense matching supported by triangular meshes , 2011 .

[17]  Robert Pless,et al.  Extrinsic calibration of a camera and laser range finder (improves camera calibration) , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[18]  Jianhui Wu,et al.  Infrared Image Area Correlation Matching Method Based on Phase Congruency , 2010, 2010 International Conference on Artificial Intelligence and Computational Intelligence.

[19]  Nasir M. Rajpoot,et al.  Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain , 2015, Pattern Recognit..

[20]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Josip Krapac,et al.  Infrared-Visual Image Registration Based on Corners and Hausdorff Distance , 2007, SCIA.

[22]  Alan F. Smeaton,et al.  Background Modelling in Infrared and Visible Spectrum Video for People Tracking , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[23]  Mohan M. Trivedi,et al.  Mutual information based registration of multimodal stereo videos for person tracking , 2007, Comput. Vis. Image Underst..

[24]  Nabil Aouf,et al.  Multimodal stereo correspondence based on phase congruency and edge histogram descriptor , 2013, Proceedings of the 16th International Conference on Information Fusion.

[25]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[26]  Shih-Schon Lin.  Review: Extending Visible Band Computer Vision Techniques to Infrared Band Images , 2001 .

[27]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[28]  Thomas B. Moeslund,et al.  Context-Aware Fusion of RGB and Thermal Imagery for Traffic Monitoring , 2016, Sensors.

[29]  Jian Zhao,et al.  Human segmentation by geometrically fusing visible-light and thermal imageries , 2012, Multimedia Tools and Applications.

[30]  Raquel Urtasun,et al.  Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation , 2014, ECCV.

[31]  Bir Bhanu,et al.  Kinematic-based human motion analysis in infrared sequences , 2002, Sixth IEEE Workshop on Applications of Computer Vision, 2002. (WACV 2002). Proceedings..

[32]  Paul Newman,et al.  Illumination Invariant Imaging : Applications in Robust Vision-based Localisation , Mapping and Classification for Autonomous Vehicles , 2014 .

[33]  Shiping Zhu,et al.  Local stereo matching algorithm with efficient matching cost and adaptive guided image filter , 2017, The Visual Computer.

[34]  Sebastian P. Kleinschmidt,et al.  Probabilistic fusion and analysis of multimodal image features , 2017, 2017 18th International Conference on Advanced Robotics (ICAR).

[35]  Uday B. Desai,et al.  Fusion of Surveillance Images in Infrared and Visible Band Using Curvelet, Wavelet and Wavelet Packet Transform , 2010, Int. J. Wavelets Multiresolution Inf. Process..