Theoretical Analysis of View Interpolation With Inaccurate Depth Information

A problem of view interpolation from a pair of rectified stereo images with inaccurate depth information is addressed. Errors in geometric information greatly affect the quality of the resulting images since inaccurate geometry causes miscorrespondences between the input images. A new theory for quantitatively analyzing the effect of depth errors and providing a principled optimization scheme based on the mean-squared error metric is proposed. The theory clarifies that, if the probabilistic distribution of the depth errors is given, an optimized view-interpolation scheme that outperforms conventional linear interpolation can be derived. It also reveals that, under specific conditions, linear interpolation is acceptable as an approximation of the optimized-interpolation scheme. Furthermore, band limitation combined with linear interpolation is also analyzed, leading to an optimal cutoff frequency, which achieves better results than the antialias scheme proposed in previous studies. Experimental results using real scenes are also presented to confirm this theory.

[1]  Andrew W. Fitzgibbon,et al.  Fields of Experts for Image-based Rendering , 2006, BMVC.

[2]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[3]  Tsuhan Chen,et al.  A survey on image-based rendering - representation, sampling and compression , 2004, Signal Process. Image Commun..

[4]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[5]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..

[6]  Andrew W. Fitzgibbon,et al.  Image-Based Rendering Using Image-Based Priors , 2005, International Journal of Computer Vision.

[7]  Bernd Girod,et al.  The Efficiency of Motion-Compensating Prediction for Hybrid Coding of Video Sequences , 1987, IEEE J. Sel. Areas Commun..

[8]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[9]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[10]  Timo Schairer,et al.  Robust non-local denoising of colored depth data , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[11]  Keita Takahashi.  Theory of Optimal View Interpolation with Depth Inaccuracy , 2010, ECCV.

[12]  Ha T. Nguyen,et al.  Error Analysis for Image-Based Rendering with Depth Information , 2009, 2006 International Conference on Image Processing.

[13]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[14]  Harry Shum,et al.  A Geometric Analysis of Light Field Rendering , 2004, International Journal of Computer Vision.

[15]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[16]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[17]  Tsuhan Chen,et al.  Spectral analysis for sampling image-based rendering data , 2003, IEEE Trans. Circuits Syst. Video Technol..

[18]  Y. Taguchi,et al.  Real-Time All-in-Focus Video-Based Rendering Using A Network Camera Array , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[19]  Toshiaki Fujii,et al.  View Generation with 3D Warping Using Depth Information for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[20]  Harry Shum,et al.  Layered lumigraph with LOD control , 2002, Comput. Animat. Virtual Worlds.

[21]  Jaejoon Lee,et al.  Edge-adaptive transforms for efficient depth map coding , 2010, 28th Picture Coding Symposium.

[22]  Masayuki Tanimoto,et al.  Multiview Imaging and 3DTV , 2007, IEEE Signal Processing Magazine.

[23]  Michael J. Black,et al.  Fields of Experts , 2009, International Journal of Computer Vision.

[24]  Richard Szeliski,et al.  High-accuracy stereo depth maps using structured light , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[25]  Bernd Girod,et al.  Rate-distortion analysis for light field coding and streaming , 2006, Signal Process. Image Commun..