Accurate real-time multi-camera stereo-matching on the GPU for 3D reconstruction

Using multi-camera matching techniques for 3d reconstruction there is usually the trade-off between the quality of the computed depth map and the speed of the computations. Whereas high quality matching methods take several seconds to several minutes to compute a depth map for one set of images, real-time methods achieve only low quality results. In this paper we present a multi-camera matching method that runs in real-time and yields high resolution depth maps. Our method is based on a novel multi-level combination of normalized cross correlation, deformed matching windows based on the multi-level depth map information, and sub-pixel precise disparity maps. The whole process is implemented completely on the GPU. With this approach we can process four 0.7 megapixel images in 129 milliseconds to a full resolution 3d depth map. Our technique is tailored for the recognition of non-technical shapes, because our target application is face recognition.

[1]  Frederic Devernay.  A Non-Maxima Suppression Method for Edge Detection with Sub-Pixel Accuracy , 1995 .

[2]  Yuichi Ohta,et al.  Occlusion detectable stereo-occlusion patterns in camera matrix , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Atsuto Maki,et al.  Stereo Matching with Direct Surface Orientation Recovery , 1998, BMVC.

[4]  Reinhard Koch,et al.  Realistic 3-D scene modeling from uncalibrated image sequences , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[5]  Don Ray Murray,et al.  Using Real-Time Stereo Vision for Mobile Robot Navigation , 2000, Auton. Robots.

[6]  O. Faugeras,et al.  Straight lines have to be straight , 2001, Machine Vision and Applications.

[7]  Kenneth Moreland,et al.  The FFT on a GPU , 2003, HWWS '03.

[8]  Ruigang Yang,et al.  Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware † , 2003, Comput. Graph. Forum.

[9]  Tsuhan Chen,et al.  A Self-Reconfigurable Camera Array , 2004, Rendering Techniques.

[10]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[11]  Ruigang Yang,et al.  A versatile stereo implementation on commodity graphics hardware , 2005, Real Time Imaging.

[12]  Jia-Ching Cheng,et al.  A real-time multiresolutional stereo matching algorithm , 2005, IEEE International Conference on Image Processing 2005.

[13]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[14]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[15]  Christopher Zach,et al.  High-Performance Multi-View Reconstruction , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[16]  Cheng Lei,et al.  Region-Tree Based Stereo Using Dynamic Programming Optimization , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[17]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Hiroshi Nakajima,et al.  A Sub-Pixel Stereo Correspondence Technique Based on 1D Phase-only Correlation , 2007, 2007 IEEE International Conference on Image Processing.

[19]  J. Ponce,et al.  Accurate, Dense, and Robust Multi-View Stereopsis , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  A. K. Lambers Posluschny,et al.  Effective High Resolution 3D Geometric Reconstruction of Heritage and Archaeological Sites from Images , 2008 .

[21]  Roberto Cipolla,et al.  Using Multiple Hypotheses to Improve Depth-Maps for Multi-View Stereo , 2008, ECCV.

[22]  Payman Moallem,et al.  Feature Based Dense Stereo Matching using Dynamic Programming and Color , 2008 .

[23]  Pascal Fua,et al.  On benchmarking camera calibration and multi-view stereo for high resolution imagery , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Duc Truong Pham,et al.  Reverse Engineering–Hardware and Software , 2008 .

[25]  Leif Kobbelt,et al.  Interactive Pixel‐Accurate Free Viewpoint Rendering from Images with Silhouette Aware Sampling , 2009, Comput. Graph. Forum.

[26]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.