Theory of Optimal View Interpolation with Depth Inaccuracy

Depth inaccuracy greatly affects the quality of free-viewpoint image synthesis. A theoretical framework for a simplified view interpolation setup to quantitatively analyze the effect of depth inaccuracy and provide a principled optimization scheme based on the mean squared error metric is proposed. The theory clarifies that if the probabilistic distribution of disparity errors is available, optimal view interpolation that outperforms conventional linear interpolation can be achieved. It is also revealed that under specific conditions, the optimal interpolation converges to linear interpolation. Furthermore, appropriate band-limitation combined with linear interpolation is also discussed, leading to an easy algorithm that achieves near-optimal quality. Experimental results using real scenes are also presented to confirm this theory.

[1]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[2]  Tsuhan Chen,et al.  A survey on image-based rendering - representation, sampling and compression , 2004, Signal Process. Image Commun..

[3]  Y. Taguchi,et al.  Real-Time All-in-Focus Video-Based Rendering Using A Network Camera Array , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[4]  Tsuhan Chen,et al.  Spectral analysis for sampling image-based rendering data , 2003, IEEE Trans. Circuits Syst. Video Technol..

[5]  Masayuki Tanimoto,et al.  Multiview Imaging and 3DTV , 2007, IEEE Signal Processing Magazine.

[6]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[7]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[8]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[9]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[10]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[11]  Bernd Girod,et al.  Rate-distortion analysis for light field coding and streaming , 2006, Signal Process. Image Commun..

[12]  Harry Shum,et al.  A Geometric Analysis of Light Field Rendering , 2004, International Journal of Computer Vision.

[13]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..

[14]  Bernd Girod,et al.  The Efficiency of Motion-Compensating Prediction for Hybrid Coding of Video Sequences , 1987, IEEE J. Sel. Areas Commun..

[15]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[16]  Harry Shum,et al.  Layered lumigraph with LOD control , 2002, Comput. Animat. Virtual Worlds.

[17]  H. Gharavi,et al.  Theoretical model and optimal prefilter for view interpolation , 2008, 2008 15th IEEE International Conference on Image Processing.