Optimization of the number of rays in interpolation for light field based free viewpoint systems

Light field (LF) rendering is widely used in free viewpoint video systems (FVV). Different methods have been proposed to employ depth maps to improve the rendering quality. However, estimation of depth is often error-prone. In this paper, a new method based on the concept of effective sampling density (ESD) is proposed for evaluating the depth-based LF rendering algorithms at different levels of errors in the depth estimation. In addition, for a given rendering quality, we provide an estimation of number of rays required in the interpolation algorithm to compensate for the adverse effect caused by errors in depth maps. The proposed method is particularly useful in designing a rendering algorithm with inaccurate knowledge of depth to achieve the required rendering quality. Both the theoretical study and numerical simulations have verified the efficacy of the proposed method.

[1]  Paul S. Fisher,et al.  Image quality measures and their performance , 1995, IEEE Trans. Commun..

[2]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[3]  Stefan Winkler.  Video quality and beyond , 2007, 2007 15th European Signal Processing Conference.

[4]  Donald S. Fussell,et al.  Uniformly Sampled Light Fields , 1998, Rendering Techniques.

[5]  Tiago Rosa Maria Paula Queluz,et al.  Towards Objective Metrics for Blind Assessment of Images Quality , 2006, 2006 International Conference on Image Processing.

[6]  Patrick Le Callet,et al.  Towards a New Quality Metric for 3-D Synthesized View Assessment , 2011, IEEE Journal of Selected Topics in Signal Processing.

[7]  Tsuhan Chen,et al.  Light Field Sampling , 2006, Light Field Sampling.

[8]  Harry Shum,et al.  Plenoptic sampling , 2000, SIGGRAPH.

[9]  Stefan Winkler,et al.  A perceptual distortion metric for digital color images , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[10]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[11]  Leonard McMillan,et al.  Dynamically reparameterized light fields , 2000, SIGGRAPH.

[12]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[13]  Farzad Safaei,et al.  A method for calculating the minimum number of cameras in a light field based free viewpoint video system , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[14]  Jean-Yves Guillemaut,et al.  Objective Quality Assessment in Free-Viewpoint Video Production , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[15]  Harry Shum,et al.  A Geometric Analysis of Light Field Rendering , 2004, International Journal of Computer Vision.

[16]  Alan C. Bovik,et al.  A Structural Similarity Metric for Video Based on Motion Models , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[17]  Pascual Capilla,et al.  Image quality metric based on multidimensional contrast perception models , 1999 .

[18]  Harry Shum,et al.  Pop-up light field: An interactive image-based modeling and rendering system , 2004, TOGS.

[19]  Tsuhan Chen,et al.  Spectral analysis for sampling image-based rendering data , 2003, IEEE Trans. Circuits Syst. Video Technol..

[20]  Farzad Safaei,et al.  A quantitative approach for comparison and evaluation of light field rendering techniques , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[21]  Heung-Yeung Shum,et al.  An Optical Analysis of Light Field Rendering , 2002 .

[22]  Amir Mousavinia,et al.  Scene-adaptive configuration of two cameras using the correspondence field function , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[23]  Leonard McMillan,et al.  Scam light field rendering , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[24]  Takeshi Naemura,et al.  Layered light-field rendering with focus measurement , 2006, Signal Process. Image Commun..

[25]  Farzad Safaei,et al.  Non-uniform sampling of plenoptic signal based on the scene complexity variations for a free viewpoint video system , 2013, 2013 IEEE International Conference on Image Processing.

[26]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[27]  Wanqing Li,et al.  Objective evaluation of light field rendering methods using effective sampling density , 2011, 2011 IEEE 13th International Workshop on Multimedia Signal Processing.

[28]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[29]  Patrick Le Callet,et al.  Can 3D synthesized views be reliably assessed through usual subjective and objective evaluation protocols? , 2011, 2011 18th IEEE International Conference on Image Processing.

[30]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[31]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[32]  Dan Zeng,et al.  An efficient method for all-in-focused light field rendering , 2010, 2010 3rd International Conference on Computer Science and Information Technology.