Real-Time Stereo Vision System: A Multi-Block Matching on GPU

Real-time stereo vision is attractive in many areas such as outdoor mapping and navigation. As a popular accelerator in the image processing field, GPU is widely used for the studies of the stereo vision algorithms. Recently, many stereo vision systems on GPU have achieved low error rate, as a result of the development of deep learning. However, their processing speed is normally far from the real-time requirement. In this paper, we propose a real-time stereo vision system on GPU for the high-resolution images. This system also maintains a low error rate compared with other fast systems. In our approach, the image is resized to reduce the computational complexity and to realize the real-time processing. The low error rate is kept by using the cost aggregation with multiple blocks, secondary matching and sub-pixel estimation. Its processing speed is 41 fps for $2888\times 1920$ pixels images when the maximum disparity is 760.

[1]  Wei Chen,et al.  Learning for Disparity Estimation Through Feature Constancy , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  Chen Yang,et al.  Real-time hardware stereo matching using guided image filter , 2016, 2016 International Great Lakes Symposium on VLSI (GLSVLSI).

[3]  Victor S. Lempitsky,et al.  End-to-End learning of cost-volume aggregation for real-time dense stereo , 2016, 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP).

[4]  Tsutomu Maruyama,et al.  Fast and Accurate Stereo Vision System on FPGA , 2014, TRETS.

[5]  Andreas Geiger,et al.  Efficient Large-Scale Stereo Matching , 2010, ACCV.

[6]  Thomas Pock,et al.  End-to-End Training of Hybrid CNN-CRF Models for Stereo , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Jiamao Li,et al.  Efficient Stereo Matching Leveraging Deep Local and Context Information , 2017, IEEE Access.

[8]  Theocharis Theocharides,et al.  High-quality real-time hardware stereo matching based on guided image filtering , 2014, 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[9]  Lifeng Sun,et al.  Binary stereo matching , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[10]  Patrice Delmas,et al.  Toward a real-time belief propagation stereo reconstruction for computers, robots, and beyond , 2015, 2015 International Conference on Image and Vision Computing New Zealand (IVCNZ).

[11]  Eric Psota,et al.  Real-Time Stereo Matching on CUDA Using an Iterative Refinement Method for Adaptive Support-Weight Correspondences , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Andreas Koch,et al.  A Scalable High-Performance Hardware Architecture for Real-Time Stereo Vision by Semi-Global Matching , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[13]  Terry M. Peters,et al.  Binocular Endoscopic 3-D Scene Reconstruction Using Color and Gradient-Boosted Aggregation Stereo Matching for Robotic Surgery , 2015, ICIG.

[14]  Haidi Ibrahim,et al.  Stereo matching algorithm based on per pixel difference adjustment, iterative guided filter and graph segmentation , 2017, J. Vis. Commun. Image Represent..

[15]  Hoi-Jun Yoo,et al.  An energy-efficient parallel multi-core ADAS processor with robust visual attention and workload-prediction DVFS for real-time HD stereo stream , 2016, 2016 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS XIX).

[16]  Cevahir Çigla.  Recursive edge-aware filters for stereo matching , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[17]  François Fleuret,et al.  Weakly Supervised Learning of Deep Metrics for Stereo Reconstruction , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[18]  Xi Wang,et al.  High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth , 2014, GCPR.

[19]  Ang Li,et al.  Coordinating Multiple Disparity Proposals for Stereo Computation , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Zhaoyang Wang,et al.  Real-time, high-accuracy 3D imaging and shape measurement. , 2015, Applied optics.

[21]  Jean-Christophe Nebel,et al.  An optimized stereo vision implementation for embedded systems: application to RGB and infra-red images , 2014, Journal of Real-Time Image Processing.

[22]  Julian Eggert,et al.  A Two-Stage Correlation Method for Stereoscopic Depth Estimation , 2010, 2010 International Conference on Digital Image Computing: Techniques and Applications.

[23]  Roy P. Paily,et al.  Design and Implementation of Blind Assistance System Using Real Time Stereo Vision Algorithms , 2016, 2016 29th International Conference on VLSI Design and 2016 15th International Conference on Embedded Systems (VLSID).

[24]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Vamshhi Pavan Kumar Varma Vegeshna,et al.  Stereo Matching with Color-Weighted Correlation, Hierachical Belief Propagation and Occlusion Handling , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[26]  Xiang Li,et al.  Accurate Dense Stereo Matching Based on Image Segmentation Using an Adaptive Multi-Cost Approach , 2016, Symmetry.

[27]  Philip J. Fleming,et al.  How not to lie with statistics: the correct way to summarize benchmark results , 1986, CACM.

[28]  Yong-Sheng Chen,et al.  Pyramid Stereo Matching Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  C. Stentoumis,et al.  On accurate dense stereo-matching using a local adaptive multi-cost approach , 2014 .

[30]  Hongyang Chao,et al.  MeshStereo: A Global Stereo Model with Mesh Alignment Regularization for View Interpolation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Miao Liao,et al.  Real-time stereo using approximated joint bilateral filtering and dynamic programming , 2012, Journal of Real-Time Image Processing.

[32]  Stefan K. Gehrig,et al.  A Flexible High-Resolution Real-Time Low-Power Stereo Vision Engine , 2015, ICVS.

[33]  John Morris,et al.  Comparison of FPGA and GPU implementations of real-time stereo vision , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[34]  Song Zhang,et al.  High-Resolution, Real-time 3D Shape Acquisition , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[35]  Richard Bowden,et al.  Stereo reconstruction using top-down cues , 2017, Comput. Vis. Image Underst..

[36]  M. Dehnavi,et al.  FPGA based real-time on-road stereo vision system , 2017, J. Syst. Archit..

[37]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[38]  Edna Barros,et al.  Hardware module for low-resource and real-time stereo vision engine using semi-global matching approach , 2017, 2017 30th Symposium on Integrated Circuits and Systems Design (SBCCI).

[39]  Byungin Moon,et al.  An accurate and cost-effective stereo matching algorithm and processor for real-time embedded multimedia systems , 2016, Multimedia Tools and Applications.

[40]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Xin Yu,et al.  3D cost aggregation with multiple minimum spanning trees for stereo matching. , 2017, Applied optics.

[42]  Li Zhang,et al.  Cascaded Multi-scale and Multi-dimension Convolutional Neural Network for Stereo Matching , 2018, 2018 IEEE Visual Communications and Image Processing (VCIP).

[43]  Alex Kendall,et al.  End-to-End Learning of Geometry and Context for Deep Stereo Regression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Madaín Pérez Patricio,et al.  An FPGA stereo matching unit based on fuzzy logic , 2016, Microprocess. Microsystems.

[45]  Philippos Mordohai,et al.  CBMV: A Coalesced Bidirectional Matching Volume for Disparity Estimation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[46]  Jae Wook Jeon,et al.  Robust non-local stereo matching for outdoor driving images using segment-simple-tree , 2015, Signal Process. Image Commun..

[47]  Julian Eggert,et al.  A multi-block-matching approach for stereo , 2015, 2015 IEEE Intelligent Vehicles Symposium (IV).

[48]  H. Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[49]  Xi Jin,et al.  A real-time global stereo-matching on FPGA , 2016, Microprocess. Microsystems.

[50]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Antonio M. López,et al.  Embedded Real-time Stereo Estimation via Semi-Global Matching on the GPU , 2016, ICCS.

[52]  Yu Wang,et al.  Real-time high-quality stereo vision system in FPGA , 2013, 2013 International Conference on Field-Programmable Technology (FPT).