Video-based character animation

In this paper we introduce a video-based representation for free viewpoint visualization and motion control of 3D character models created from multiple view video sequences of real people. Previous approaches to video-based rendering provide no control of scene dynamics to manipulate, retarget, and create new 3D content from captured scenes. Here we contribute a new approach, combining image based reconstruction and video-based animation to allow controlled animation of people from captured multiple view video sequences. We represent a character as a motion graph of free viewpoint video motions for animation control. We introduce the use of geometry videos to represent reconstructed scenes of people for free viewpoint video rendering. We describe a novel spherical matching algorithm to derive global surface to surface correspondence in spherical geometry images for motion blending and the construction of seamless transitions between motion sequences. Finally, we demonstrate interactive video-based character animation with real-time rendering and free viewpoint visualization. This approach synthesizes highly realistic character animations with dynamic surface shape and appearance captured from multiple view video of people.

[1]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[2]  David A. Forsyth,et al.  Motion synthesis from annotations , 2003, ACM Trans. Graph..

[3]  Adrian Hilton,et al.  Virtual view synthesis of people from multiple view video sequences , 2005, Graph. Model..

[4]  Pedro V. Sander,et al.  Geometry videos: a new representation for 3D animations , 2003, SCA '03.

[5]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[6]  Thomas Malzbender,et al.  Generalized Voxel Coloring , 1999, Workshop on Vision Algorithms.

[7]  Irfan A. Essa,et al.  Controlled animation of video sprites , 2002, SCA '02.

[8]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[9]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[10]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[11]  Hujun Bao,et al.  3D surface filtering using spherical harmonics , 2004, Comput. Aided Des..

[12]  Hugues Hoppe,et al.  Progressive meshes , 1996, SIGGRAPH.

[13]  Hans Peter Graf,et al.  Photo-Realistic Talking-Heads from Image Samples , 2000, IEEE Trans. Multim..

[14]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[15]  Adrian Hilton,et al.  Model-based multiple view reconstruction of people , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[16]  Pascal Fua,et al.  Articulated Soft Objects for Video-based Body Modeling , 2001, ICCV.

[17]  K HodginsJessica,et al.  Interactive control of avatars animated with human motion data , 2002 .

[18]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[19]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[20]  Hugues Hoppe,et al.  Spherical parametrization and remeshing , 2003, ACM Trans. Graph..

[21]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, ACM Trans. Graph..

[22]  Marcus A. Magnor,et al.  Space-time isosurface evolution for temporally coherent 3D reconstruction , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[23]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[24]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[25]  Anand Rangarajan,et al.  A new point matching algorithm for non-rigid registration , 2003, Comput. Vis. Image Underst..

[26]  Hugues Hoppe,et al.  Spherical parametrization and remeshing , 2003, ACM Trans. Graph..

[27]  Harry Shum,et al.  Motion texture: a two-level statistical model for character motion synthesis , 2002, ACM Trans. Graph..

[28]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH '08.

[29]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[30]  Takeo Kanade,et al.  Spatio-Temporal View Interpolation , 2002, Rendering Techniques.

[31]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[32]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[33]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[34]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.