Optimal Representation of Multiple View Video

Multi-view video acquisition is widely used for reconstruction and free-viewpoint rendering of dynamic scenes by directly resampling from the captured images. This paper addresses the problem of optimally resampling and representing multi-view video to obtain a compact representation without loss of the view-dependent dynamic surface appearance. Spatio-temporal optimisation of the multi-view resampling is introduced to extract a coherent multi-layer texture map video. This resampling is combined with a surface-based optical flow alignment between views to correct for errors in geometric reconstruction and camera calibration which result in blurring and ghosting artefacts. The multi-view alignment and optimised resampling results in a compact representation with minimal loss of information allowing high-quality free-viewpoint rendering. Evaluation is performed on multi-view datasets for dynamic sequences of cloth, faces and people. The representation achieves >90% compression without significant loss of visual quality.

[1]  Vagia Tsiminaki,et al.  High Resolution 3D Shape Texture from Multiple Videos , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[3]  Slobodan Ilic,et al.  Probabilistic Deformable Surface Tracking from Multiple Videos , 2010, ECCV.

[4]  Jean-Philippe Pons,et al.  Spatio-temporal image-based texture atlases for dynamic 3-D models , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[5]  Adrian Hilton,et al.  Model-based multiple view reconstruction of people , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[6]  Adrian Hilton,et al.  4D video textures for interactive character appearance , 2014, Comput. Graph. Forum.

[7]  Jean-Philippe Pons,et al.  Seamless image-based texture atlases using multi-band blending , 2008, 2008 19th International Conference on Pattern Recognition.

[8]  Adrian Hilton,et al.  A Free-Viewpoint Video Renderer , 2009, J. Graphics, GPU, & Game Tools.

[9]  Daniel Cremers,et al.  Superresolution texture maps for multiview reconstruction , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[10]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[11]  Martin Klaudiny,et al.  Towards Optimal Non-rigid Surface Tracking , 2012, ECCV.

[12]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[13]  Martin Klaudiny,et al.  Global Non-rigid Alignment of Surface Sequences , 2013, International Journal of Computer Vision.

[14]  Pushmeet Kohli,et al.  Reduce, reuse & recycle: Efficiently solving multi-label MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Pushmeet Kohli,et al.  Unwrap mosaics: a new representation for video editing , 2008, SIGGRAPH 2008.

[16]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[17]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[18]  Dan Piponi,et al.  Seamless texture mapping of subdivision surfaces by model pelting and texture blending , 2000, SIGGRAPH.

[19]  Victor S. Lempitsky,et al.  Seamless Mosaicing of Image-Based Texture Maps , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, SIGGRAPH 2008.

[21]  Pascal Monasse,et al.  Multi-view Texturing of Imprecise Mesh , 2009, ACCV.

[22]  Daniel Cohen-Or,et al.  Seamless Montage for Texturing Models , 2010, Comput. Graph. Forum.

[23]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.