Towards Real-time Stereo using Non-uniform Image Sampling and Sparse Dynamic Programming

Constructing the 3D mesh of a scene from stereo images is a major task in computer vision. It usually involves several steps including stereo matching and meshing. Unfortunately, the time required to generate the 3D mesh is time demanding due to the large amount of pixels to be processed. In this work, we propose a framework to accelerate the overall process. The key issue is to first reduce the number of pixels by approximating an image with a content adaptive mesh. The nodes of the mesh are sparse and they represent the non-uniform samples of the image. To benefit from the reduced set of pixels, we formulate a dynamic programming based stereo matching algorithm which computes the depth only at the sparse samples. We then show by setting up some tests using some real images that the non-uniform samples are sufficient to recover the original dense depth map of the scene by interpolating them using the mesh. The results obtained also show that the employment of the proposed strategy reduces the overall processing time of stereo matching to more than 50% of the original time. We are now able to construct scenes in real-time using less computational resources.

[1]  Radim Sára,et al.  Finding the Largest Unambiguous Component of Stereo Matching , 2002, ECCV.

[2]  Ruigang Yang,et al.  Improved Real-Time Stereo on Commodity Graphics Hardware , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[3]  Miguel Ángel García,et al.  Coarse-to-fine approximation of range images with bounded error adaptive triangular meshes , 2007, J. Electronic Imaging.

[4]  Ruigang Yang,et al.  Image-gradient-guided real-time stereo on graphics hardware , 2005, Fifth International Conference on 3-D Digital Imaging and Modeling (3DIM'05).

[5]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Hans-Peter Seidel,et al.  An integrating approach to meshing scattered point data , 2005, SPM '05.

[7]  Radim Sára,et al.  Stratified Dense Matching for Stereopsis in Complex Scenes , 2003, BMVC.

[8]  Giovanni Ramponi,et al.  An adaptive irregular sampling algorithm and its application to image coding , 2001, Image Vis. Comput..

[9]  S. Wille.  A structured tri‐tree search method for generation of optimal unstructured finite element grids in two and three dimensions , 1992 .

[10]  H. Hirschmüller.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Stereo Processing by Semi-global Matching and Mutual Information , 2022 .

[11]  Kostas Daniilidis,et al.  Stereo-based environment scanning for immersive telepresence , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  Jonathan M. Garibaldi,et al.  Real-Time Correlation-Based Stereo Vision with Reduced Border Errors , 2002, International Journal of Computer Vision.

[13]  Anthony Vetro,et al.  Use of two-dimensional deformable mesh structures for video coding. II. The analysis problem and a region-based coder employing an active mesh representation , 1996, IEEE Trans. Circuits Syst. Video Technol..

[14]  Savitri Bevinakoppa,et al.  DIGITAL IMAGE COMPRESSION TECHNIQUES , 2014 .

[15]  Takeo Kanade,et al.  Stereo by Intra- and Inter-Scanline Search Using Dynamic Programming , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[17]  Carlo Tomasi,et al.  Depth Discontinuities by Pixel-to-Pixel Stereo , 1999, International Journal of Computer Vision.

[18]  Changming Sun,et al.  Fast Stereo Matching by Iterated Dynamic Programming and Quadtree Subregioning , 2004, BMVC.

[19]  R. Pajarola.  Overview of Quadtree-based Terrain Triangulation and Visualization , 2002 .

[20]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[21]  Miao Liao,et al.  Real-time Global Stereo Matching Using Hierarchical Belief Propagation , 2006, BMVC.

[22]  Minglun Gong,et al.  Near real-time reliable stereo matching using programmable graphics hardware , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[23]  James F. O'Brien,et al.  Interpolating and approximating implicit surfaces from polygon soup , 2004, SIGGRAPH Courses.

[24]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Olga Veksler,et al.  Stereo correspondence by dynamic programming on a tree , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[26]  Yongyi Yang,et al.  A fast approach for accurate content-adaptive mesh generation , 2003, IEEE Trans. Image Process..

[27]  Markus H. Gross,et al.  Fast Multiresolution Surface Meshing , 1995, IEEE Visualization.

[28]  Miao Liao,et al.  High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[29]  Georg Passig,et al.  A Photorealistic Predictive Display , 2004, Presence: Teleoperators & Virtual Environments.