Algorithm and Architecture of Disparity Estimation With Mini-Census Adaptive Support Weight

High-performance real-time stereo vision system is crucial to various stereo vision applications, such as robotics, autonomous vehicles, multiview video coding, freeview TV, and 3-D video conferencing. In this paper, we proposed a high-performance hardware-friendly disparity estimation algorithm called mini-census adaptive support weight (MCADSW) and also proposed its corresponding real-time very large scale integration (VLSI) architecture. To make the proposed MCADSW algorithm hardware-friendly, we proposed simplification techniques such as using mini-census, removing proximity weight, using YUV color representation, using Manhattan color distance, and using scaled-and-truncate weight approximation. After applied these simplifications, the MCADSW algorithm was not only hardware-friendly, but was also 1.63 times faster. In the corresponding real-time VLSI architecture, we proposed partial column reuse and access reduction with expanded window to significantly reduce the bandwidth requirement. The proposed architecture was implemented using United Microelectronics Corporation (UMC) 90 nm complementary metal-oxide-semiconductor technology and can achieve a disparity estimation frame rate of 42 frames/s for common intermediate format size images when clocked at 95 MHz. The synthesized gate-count and memory size is 563 k and 21.3 kB, respectively.

[1]  Heiko Hirschmüller,et al.  Improvements in real-time correlation-based stereo vision , 2001, CVPR 2001.

[2]  Takeo Kanade,et al.  A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Masanori Hariyama,et al.  Reliable stereo matching for highly-safe intelligent vehicles and its VLSI implementation , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[4]  Masanori Hariyama,et al.  FPGA implementation of a stereo matching processor based on window-parallel-and-pixel-parallel architecture , 2005 .

[5]  R. D. Arnold.  Automated stereo perception , 1983 .

[6]  Tomoharu Nakahara,et al.  A Multiple-baseline Stereo Method , 2002 .

[7]  Jonathan M. Garibaldi,et al.  Real-Time Correlation-Based Stereo Vision with Reduced Border Errors , 2002, International Journal of Computer Vision.

[8]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[9]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[10]  H. Hirschmüller.  Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information , 2005, CVPR.

[11]  Ruigang Yang,et al.  Image-gradient-guided real-time stereo on graphics hardware , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[12]  Laurent Moll,et al.  Real time correlation-based stereo: algorithm, implementations and applications , 1993 .

[13]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Gauthier Lafruit,et al.  Fast Variable Center-Biased Windowing for High-Speed Stereo on Programmable Graphics Hardware , 2007, 2007 IEEE International Conference on Image Processing.

[15]  Takeo Kanade,et al.  A locally adaptive window for signal matching , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[16]  Tian-Sheuan Chang,et al.  Analysis of color space and similarity measure impact on stereo block matching , 2008, APCCAS 2008 - 2008 IEEE Asia Pacific Conference on Circuits and Systems.

[17]  Federico Tombari,et al.  Classification and evaluation of cost aggregation methods for stereo correspondence , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  H. K. Nishihara.  Real-time stereo- and motion-based figure ground discrimination and tracking using LOG sign correlation , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[19]  Tian-Sheuan Chang,et al.  Data reuse analysis of local stereo matching , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[20]  Alfred Schmitt,et al.  Real-Time Stereo by using Dynamic Programming , 2003, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[21]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Peter I. Corke,et al.  Quantitative Evaluation of Matching Methods and Validity Measures for Stereo Vision , 2001, Int. J. Robotics Res..

[23]  Masanori Hariyama,et al.  Architecture of a stereo matching VLSI processor based on hierarchically parallel memory access , 2004 .

[24]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[25]  Richard Szeliski,et al.  Handling occlusions in dense multi-view stereo , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[26]  Kurt Konolige,et al.  Small Vision Systems: Hardware and Implementation , 1998 .

[27]  Philippe Bekaert,et al.  Local Stereo Matching with Segmentation-based Outlier Rejection , 2006, The 3rd Canadian Conference on Computer and Robot Vision (CRV'06).

[28]  Aaron F. Bobick,et al.  Large Occlusion Stereo , 1999, International Journal of Computer Vision.

[29]  Peter Corke,et al.  Real-time stereopsis using FPGAs , 1997, TENCON '97 Brisbane - Australia. Proceedings of IEEE TENCON '97. IEEE Region 10 Annual Conference. Speech and Image Technologies for Computing and Telecommunications (Cat. No.97CH36162).

[30]  Federico Tombari,et al.  Near real-time stereo based on effective cost aggregation , 2008, 2008 19th International Conference on Pattern Recognition.

[31]  W. James MacLean,et al.  A Real-Time Large Disparity Range Stereo-System Using FPGAs , 2006, ACCV.

[32]  Yasuhiro Kobayashi,et al.  FPGA implementation of a stereo matching processor based on window-parallel-and-pixel-parallel architecture , 2005, 48th Midwest Symposium on Circuits and Systems, 2005..

[33]  Tian-Sheuan Chang,et al.  Real-Time DSP Implementation on Local Stereo Matching , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[34]  Takeo Kanade,et al.  CMU Video-Rate Stereo Machine , 1995 .

[35]  Olga Veksler.  Stereo Matching by Compact Windows via Minimum Ratio Cycle , 2001, ICCV.

[36]  S. Chambon,et al.  EVALUATION OF DIFFERENT METHODS FOR USING COLOUR INFORMATION IN GLOBAL STEREO MATCHING APPROACHES , 2008 .

[37]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  John Woodfill,et al.  Real-time stereo vision on the PARTS reconfigurable computer , 1997, Proceedings. The 5th Annual IEEE Symposium on Field-Programmable Custom Computing Machines Cat. No.97TB100186).

[39]  Chao Chen,et al.  VLSI Architecture for MRF Based Stereo Matching , 2007, SAMOS.

[40]  Miao Liao,et al.  Real-time Global Stereo Matching Using Hierarchical Belief Propagation , 2006, BMVC.

[41]  Minglun Gong,et al.  Near real-time reliable stereo matching using programmable graphics hardware , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[42]  Gauthier Lafruit,et al.  Real-Time Stereo Correspondence using a Truncated Separable Laplacian Kernel Approximation on Graphics Hardware , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[43]  Miao Liao,et al.  High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[44]  Zheng Zhi.  A Region Based Stereo Matching Algorithm Using Cooperative Optimization , 2009 .

[45]  In-So Kweon,et al.  Adaptive Support-Weight Approach for Correspondence Search , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Shigeru Kimura,et al.  A convolver-based real-time stereo machine (SAZAN) , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[47]  Federico Tombari,et al.  Segmentation-Based Adaptive Support for Accurate Stereo Correspondence , 2007, PSIVT.

[48]  Hong Jeong,et al.  Real-time Stereo Vision FPGA Chip with Low Error Rate , 2007, 2007 International Conference on Multimedia and Ubiquitous Engineering (MUE'07).