An FPGA-based processing pipeline for high-definition stereo video

This paper presents a real-time processing platform for high-definition stereo video. The system is capable to process stereo video streams at resolutions up to 1, 920 × 1, 080 at 30 frames per second (1080p30). In the hybrid FPGA-GPU-CPU system, a high-density FPGA is used not only to perform the low-level image processing tasks such as color interpolation and cross-image color correction, but also to carry out radial undistortion, image rectification, and disparity estimation. We show how the corresponding algorithms can be implemented very efficiently in programmable hardware, relieving the GPU from the burden of these tasks. Our FPGA implementation results are compared with corresponding GPU implementations and with other implementations reported in the literature.

[1]  S. Nedevschi,et al.  LUT-based Image Rectification Module Implemented in FPGA , 2007, 2007 IEEE International Conference on Intelligent Computer Communication and Processing.

[2]  P. Eisert,et al.  Joint Estimation of Epipolar Geometry and Rectification Parameters using Point Correspondences for Stereoscopic TV Sequences , 2010 .

[3]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[4]  Horst Bischof,et al.  Hierarchical Disparity Estimation with Programmable Graphics Hardware , 2004, WSCG.

[5]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[6]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..

[7]  W.E. Snyder,et al.  Color image processing pipeline , 2005, IEEE Signal Processing Magazine.

[8]  A. C. Sonmez,et al.  FPGA design and implementation of a real-time stereo vision system , 2012, 2012 International Symposium on Innovations in Intelligent Systems and Applications.

[9]  Georgy Gimel'farb,et al.  Breaking the ‘Ton’: Achieving 1% depth accuracy from stereo in real time , 2009, 2009 24th International Conference Image and Vision Computing New Zealand.

[10]  Shun'ichi Kaneko,et al.  A real-time stereo vision sensor based on FPGA realization of orientation code matching , 2010, 2010 International Symposium on Optomechatronic Technologies.

[11]  David R. Bull,et al.  Projective image restoration using sparsity regularization , 2013, 2013 IEEE International Conference on Image Processing.

[12]  Bernard Mendiburu,et al.  3D Movie Making: Stereoscopic Digital Cinema from Script to Screen , 2009 .

[13]  Aljoscha Smolic,et al.  Computational stereo camera system with programmable control loop , 2011, ACM Trans. Graph..

[14]  Stefan K. Gehrig,et al.  A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching , 2009, ICVS.

[15]  Stephen Longfield,et al.  A Parameterized Stereo Vision Core for FPGAs , 2009, 2009 17th IEEE Symposium on Field Programmable Custom Computing Machines.

[16]  Horst Bischof,et al.  Hierarchical Disparity Estimation with Programmable 3D Hardware , 2004 .

[17]  Paul F. Whelan,et al.  Projective rectification from the fundamental matrix , 2005, Image Vis. Comput..

[18]  Peter Pirsch,et al.  Real-time stereo vision system using semi-global matching disparity estimation: Architecture and FPGA-implementation , 2010, 2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation.

[19]  Laurent Jacques,et al.  The PANOPTIC Camera: A Plenoptic Sensor with Real-Time Omnidirectional Capability , 2013, J. Signal Process. Syst..

[20]  Tsutomu Maruyama,et al.  A Real-Time Stereo Vision System with FPGA , 2003, FPL.

[21]  Peter Eisert,et al.  The Stereoscopic Analyzer — An image-based assistance tool for stereo shooting and 3D production , 2010, 2010 IEEE International Conference on Image Processing.

[22]  Dean Brown,et al.  Decentering distortion of lenses , 1966 .

[23]  Ioannis Andreadis,et al.  A Real-Time Occlusion Aware Hardware Structure for Disparity Map Computation , 2009, ICIAP.

[24]  Henrique S. Malvar,et al.  High-quality linear interpolation for demosaicing of Bayer-patterned color images , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[25]  Fatih Murat Porikli,et al.  Inter-camera color calibration by correlation model function , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[26]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[27]  John Morris,et al.  Comparison of FPGA and GPU implementations of real-time stereo vision , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.