4D video textures for interactive character appearance

4D Video Textures (4DVT) introduce a novel representation for rendering video‐realistic interactive character animation from a database of 4D actor performance captured in a multiple camera studio. 4D performance capture reconstructs dynamic shape and appearance over time but is limited to free‐viewpoint video replay of the same motion. Interactive animation from 4D performance capture has so far been limited to surface shape only. 4DVT is the final piece in the puzzle enabling video‐realistic interactive animation through two contributions: a layered view‐dependent texture map representation which supports efficient storage, transmission and rendering from multiple view video capture; and a rendering approach that combines multiple 4DVT sequences in a parametric motion space, maintaining video quality rendering of dynamic surface appearance whilst allowing high‐level interactive control of character motion and viewpoint. 4DVT is demonstrated for multiple characters and evaluated both quantitatively and through a user‐study which confirms that the visual quality of captured video is maintained. The 4DVT representation achieves >90% reduction in size and halves the rendering cost.

[1]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[2]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, ACM Trans. Graph..

[3]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[4]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[5]  Atsushi Nakazawa,et al.  Human video textures , 2009, I3D '09.

[6]  Michael Gleicher,et al.  Parametric motion graphs , 2007, SI3D.

[7]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[8]  Adrian Hilton,et al.  Video-based character animation , 2005, SCA '05.

[9]  Hans-Peter Seidel,et al.  Video-based characters: creating new human performances from a multi-view video database , 2011, ACM Trans. Graph..

[10]  Martin Klaudiny,et al.  Global Non-rigid Alignment of Surface Sequences , 2013, International Journal of Computer Vision.

[11]  Qionghai Dai,et al.  Capturing Relightable Human Performances under General Uncontrolled Illumination , 2013, Comput. Graph. Forum.

[12]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH '08.

[13]  Hans-Peter Seidel,et al.  MovieReshape: tracking and reshaping of humans in videos , 2010, ACM Trans. Graph..

[14]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[15]  Pushmeet Kohli,et al.  Reduce, reuse & recycle: Efficiently solving multi-label MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[17]  Hans-Peter Seidel,et al.  Shading-based dynamic shape refinement from multi-view video under general illumination , 2011, 2011 International Conference on Computer Vision.

[18]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[19]  Hans-Peter Seidel,et al.  Seeing People in Different Light — Joint Shape , Motion , and Reflectance Capture , 2007 .

[20]  Slobodan Ilic,et al.  Free-form mesh tracking: A patch-based approach , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[21]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[22]  Björn Stenger,et al.  Video Normals from Colored Lights , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Adrian Hilton,et al.  Human motion synthesis from 3D video , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Jean-Yves Guillemaut,et al.  Interactive Animation of 4D Performance Capture , 2013, IEEE Transactions on Visualization and Computer Graphics.

[25]  Anna Hilsmann,et al.  Pose Space Image Based Rendering , 2013, Comput. Graph. Forum.

[26]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[27]  Nancy Argüelles,et al.  Author ' s , 2008 .

[28]  Daniel Cohen-Or,et al.  Seamless Montage for Texturing Models , 2010, Comput. Graph. Forum.

[29]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..