A Real-Time View-Dependent Shape Optimization for High Quality Free-Viewpoint Rendering of 3D Video

This paper is aimed at proposing a new high quality free-viewpoint rendering algorithm of 3D video. The main challenge on visualizing 3D video is how to utilize the original multi-view images used to estimate the 3D surface, and how to manage the mismatches between them due to calibration and reconstruction errors. The key idea to solve this problem is to optimize the 3D shape on a per-viewpoint basis on the fly. Given a virtual viewpoint for visualization, our algorithm optimizes the 3D shape so as to maximize the photo-consistency over the surface visible from the virtual viewpoint. An evaluation demonstrates that our method outperforms the state-of-the-art rendering qualitatively and quantitatively.

[1]  Adrian Hilton,et al.  Harmonised Texture Mapping , 2010 .

[2]  Thomas Wiegand,et al.  3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[3]  Daniel Cremers,et al.  Superresolution texture maps for multiview reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[4]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[5]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[6]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[7]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[8]  Adrian Hilton,et al.  The Multiple-Camera 3-D Production Studio , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[9]  Y. Taguchi,et al.  Real-Time All-in-Focus Video-Based Rendering Using A Network Camera Array , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[10]  Pedro F. Felzenszwalb,et al.  Efficient belief propagation for early vision , 2004, CVPR 2004.

[11]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Yuichi Iwadate,et al.  Method of 3D reconstruction using graph cuts, and its application to preserving intangible cultural heritage , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[13]  Jean Ponce,et al.  Accurate Camera Calibration from Multi-View Stereo and Bundle Adjustment , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[15]  Adrian Hilton,et al.  Proj ective Surface Refinement for Free-Viewpoint Video , 2006 .

[16]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[17]  Adrian Hilton,et al.  Exact View-Dependent Visual Hulls , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[18]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[19]  Takashi Matsuyama,et al.  Simultaneous super-resolution and 3D video using graph-cuts , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Xiaojun Wu,et al.  Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video , 2004, Comput. Vis. Image Underst..

[21]  J. Starck,et al.  A Robust Free-Viewpoint Video System for Sport Scenes , 2007, 2007 3DTV Conference.

[22]  Miller,et al.  Interactive Free-Viewpoint Video , 2006 .

[23]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[24]  Henry Fuchs,et al.  Fast constructive-solid geometry display in the pixel-powers graphics system , 1986, SIGGRAPH.

[25]  Jérémie Allard,et al.  The GrImage Platform: A Mixed Reality Environment for Interactions , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[26]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[27]  Toshiaki Fujii,et al.  Free-Viewpoint TV , 2011, IEEE Signal Processing Magazine.

[28]  J. Collomosse,et al.  4D video textures for interactive character appearance , 2014, Comput. Graph. Forum.