Video-based character animation

In this paper we introduce a video-based representation for free viewpoint visualization and motion control of 3D character models created from multiple view video sequences of real people. Previous approaches to video-based rendering provide no control of scene dynamics to manipulate, retarget, and create new 3D content from captured scenes. Here we contribute a new approach, combining image based reconstruction and video-based animation to allow controlled animation of people from captured multiple view video sequences. We represent a character as a motion graph of free viewpoint video motions for animation control. We introduce the use of geometry videos to represent reconstructed scenes of people for free viewpoint video rendering. We describe a novel spherical matching algorithm to derive global surface to surface correspondence in spherical geometry images for motion blending and the construction of seamless transitions between motion sequences. Finally, we demonstrate interactive video-based character animation with real-time rendering and free viewpoint visualization. This approach synthesizes highly realistic character animations with dynamic surface shape and appearance captured from multiple view video of people.

[1]  Harry Shum,et al.  Motion texture: a two-level statistical model for character motion synthesis , 2002, ACM Trans. Graph..

[2]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[3]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[4]  David A. Forsyth,et al.  Motion synthesis from annotations , 2003, ACM Trans. Graph..

[5]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[6]  Marcus A. Magnor,et al.  Space-time isosurface evolution for temporally coherent 3D reconstruction , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[7]  Hugues Hoppe,et al.  Progressive meshes , 1996, SIGGRAPH.

[8]  Thomas Malzbender,et al.  Generalized Voxel Coloring , 1999, Workshop on Vision Algorithms.

[9]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[10]  Hujun Bao,et al.  3D surface filtering using spherical harmonics , 2004, Comput. Aided Des..

[11]  Adrian Hilton,et al.  Virtual view synthesis of people from multiple view video sequences , 2005, Graph. Model..

[12]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[13]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[14]  Hans Peter Graf,et al.  Photo-Realistic Talking-Heads from Image Samples , 2000, IEEE Trans. Multim..

[15]  Adrian Hilton,et al.  Model-based multiple view reconstruction of people , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[16]  Pedro V. Sander,et al.  Geometry videos: a new representation for 3D animations , 2003, SCA '03.

[17]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[18]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[19]  Anand Rangarajan,et al.  A new point matching algorithm for non-rigid registration , 2003, Comput. Vis. Image Underst..

[20]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH '08.

[21]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, ACM Trans. Graph..

[22]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[23]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[24]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[25]  Irfan A. Essa,et al.  Controlled animation of video sprites , 2002, SCA '02.

[26]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[27]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[28]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[29]  R. Plankers,et al.  Articulated soft objects for video-based body modeling , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.