Real-Time High-Quality Stereo Vision System in FPGA

Stereo vision is a well-known technique for acquiring depth information. In this paper, we propose a real-time high-quality stereo vision system in field-programmable gate array (FPGA). Using absolute difference-census cost initialization, cross-based cost aggregation, and semiglobal optimization, the system provides high-quality depth results for high-definition images. This is the first complete real-time hardware system that supports both cost aggregation on variable support regions and semiglobal optimization in FPGAs. Furthermore, the system is designed to be scaled with image resolution, disparity range, and parallelism degree for maximum parallel efficiency. We present the depth map quality on the Middlebury benchmark and some real-world scenarios with different image resolutions. The results show that our system performs the best among FPGA-based stereo vision systems and its accuracy is comparable with those of current top-performing software implementations. The first version of the system was demonstrated on an Altera Stratix-IV FPGA board, processing 1024 × 768 pixel images with 96 disparity levels at 67 frames/s. The system is then scaled up on a new Altera Stratix-V FPGA and the processing ability is enhanced to 1600 × 1200 pixel images with 128 disparity levels at 42 frames/s.

[1]  Miao Liao,et al.  Real-time Global Stereo Matching Using Hierarchical Belief Propagation , 2006, BMVC.

[2]  Hong Jeong,et al.  Real-time Stereo Vision FPGA Chip with Low Error Rate , 2007, 2007 International Conference on Multimedia and Ubiquitous Engineering (MUE'07).

[3]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[4]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Andreas Steininger,et al.  SAD-Based Stereo Matching Using FPGAs , 2009 .

[6]  Stefan K. Gehrig,et al.  A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching , 2009, ICVS.

[7]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[8]  Miao Liao,et al.  High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[9]  Gauthier Lafruit,et al.  Cross-Based Local Stereo Matching Using Orthogonal Integral Images , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  A. C. Sonmez,et al.  FPGA design and implementation of a real-time stereo vision system , 2012, 2012 International Symposium on Innovations in Intelligent Systems and Applications.

[12]  Sek M. Chai,et al.  Multi-Resolution Real-Time Dense Stereo Vision Processing in FPGA , 2012, 2012 IEEE 20th International Symposium on Field-Programmable Custom Computing Machines.

[13]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[14]  Tsutomu Maruyama,et al.  A fast and high quality stereo matching algorithm on FPGA , 2012, 22nd International Conference on Field Programmable Logic and Applications (FPL).

[15]  S. Sabihuddin,et al.  Dynamic programming approach to high frame-rate stereo correspondence: A pipelined architecture implemented on a field programmable gate array , 2008, 2008 Canadian Conference on Electrical and Computer Engineering.

[16]  Tsutomu Maruyama,et al.  Fast and Accurate Stereo Vision System on FPGA , 2014, TRETS.

[17]  Yuchen Hao,et al.  FPGA based memory efficient high resolution stereo vision system for video tolling , 2012, 2012 International Conference on Field-Programmable Technology.

[18]  Ruigang Yang,et al.  Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware † , 2003, Comput. Graph. Forum.

[19]  Diederik Verkest,et al.  Real-time high-definition stereo matching on FPGA , 2011, FPGA '11.

[20]  HirschmullerHeiko.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008 .

[21]  Xing Mei,et al.  On building an accurate stereo matching system on graphics hardware , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[22]  Philip L. Davidson,et al.  Real-time stereo vision using semi-global matching on programmable graphics hardware , 2006, SIGGRAPH '06.

[23]  Peter Pirsch,et al.  Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation , 2010, 2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation.

[24]  Xu Chen,et al.  Hardware Acceleration for Accurate Stereo Vision System using Mini-Census Adaptive Support Region , 2013 .

[25]  Tsutomu Maruyama,et al.  A real-time stereo vision system using a tree-structured dynamic programming on FPGA , 2012, FPGA '12.

[26]  Yu Wang,et al.  Real-time high-quality stereo vision system in FPGA , 2013, 2013 International Conference on Field-Programmable Technology (FPT).