GPU-enhanced Multimodal Dense Matching

Multiple modalities for stereo matching are beneficial for robust path estimation and actioning of autonomous robots in harsh environments, e.g. in the presence of smoke and dust. In order to combine the information resulting from the different modalities, a dense stereo matching approach based on semi-global matching and a combined cost function using cross-based support regions and phase congruency shows a good performance. However, these computationally complex algorithmic steps set high requirements for the mobile processing platform and prohibit a real-time execution at limited power budget on mobile platforms. Therefore, this paper explores the usage of graphic processors for the parallelization and acceleration of the aforementioned algorithm. The resulting implementation performs the computation of phase congruency and cross-based support regions at 68 and 5 frames per second for $[960\mathrm{x}560]$ pixel images on a Nvidia Quadro P5000 and Tegra X2 GPU respectively.

[1]  Stijn De Beugher,et al.  The Battle of the Giants - A Case Study of GPU vs FPGA Optimisation for Real-time Image Processing , 2014, PECCS.

[2]  Rudy Lauwereins,et al.  Real-time accurate stereo with bitwise fast voting on CUDA , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[3]  Clemens Rabe,et al.  Real-time Semi-Global Matching on the CPU , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[4]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[5]  Christian Heipke,et al.  Multimodal Dense Stereo Matching , 2018, GCPR.

[6]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Sebastian P. Kleinschmidt,et al.  Probabilistic fusion and analysis of multimodal image features , 2017, 2017 18th International Conference on Advanced Robotics (ICAR).

[8]  Sergiu Nedevschi,et al.  SORT-SGM: Subpixel Optimized Real-Time Semiglobal Matching for Intelligent Vehicles , 2012, IEEE Transactions on Vehicular Technology.

[9]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[10]  Peter Kovesi,et al.  Image Features from Phase Congruency , 1995 .

[11]  Ines Ernst,et al.  Mutual Information Based Semi-Global Stereo Matching on the GPU , 2008, ISVC.

[12]  C. Heipke,et al.  Illumination Invariant Dense Image Matching based on Sparse Features , 2018 .

[13]  Gauthier Lafruit,et al.  Real-time stereo matching: A cross-based local approach , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  Sergiu Nedevschi,et al.  GPU optimization of the SGM stereo algorithm , 2010, Proceedings of the 2010 IEEE 6th International Conference on Intelligent Computer Communication and Processing.

[15]  Peter Kovesi,et al.  Phase Congruency Detects Corners and Edges , 2003, DICTA.

[16]  Sebastian P. Kleinschmidt,et al.  Visual Multimodal Odometry: Robust Visual Odometry in Harsh Environments , 2018, 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR).

[17]  Peter Pirsch,et al.  Real-time semi-global matching disparity estimation on the GPU , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[18]  Oge Marques,et al.  Stereo depth with a Unified Architecture GPU , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[19]  Michael Garland,et al.  Designing efficient sorting algorithms for manycore GPUs , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[20]  Gauthier Lafruit,et al.  Cross-Based Local Stereo Matching Using Orthogonal Integral Images , 2009, IEEE Transactions on Circuits and Systems for Video Technology.