Free-viewpoint Video for Interactive Character Animation

Conventional 2D video production provides a fixed-view reproduction of a recorded event in which a director defines every moment to be viewed. Multiple camera studio systems are now being developed to capture real-world events from multiple viewpoints to replay the event in 3D as a freeviewpoint video. Free-viewpoint video provides the consumer the flexibility to interact with a scene by changing the viewpoint in replaying the event. The challenge in computer vision and computer graphics remains to capture real world events for both replay and to provide the director or consumer the freedom to synthesize new unseen content. In this paper we explore the use of free-viewpoint video in content synthesis by re-using captured scenes of people. Our work combines image based scene reconstruction and video-based animation techniques to provide a new approach that allows interactive character animation from free-viewpoint video.

[1]  Irfan A. Essa,et al.  Controlled animation of video sprites , 2002, SCA '02.

[2]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[3]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[4]  Masayuki Tanimoto.  Free Viewpoint Television (FTV) , 2007 .

[5]  O. Faugeras,et al.  Variational principles, surface evolution, PDE's, level set methods and the stereo problem , 1998, 5th IEEE EMBS International Summer School on Biomedical Imaging, 2002..

[6]  Oliver Grau,et al.  Real-Time Production and Delivery of 3D Media , 2002 .

[7]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[8]  Adrian Hilton,et al.  Volumetric Stereo with Silhouette and Feature Constraints , 2006, BMVC.

[9]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[10]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH '08.

[11]  Olivier D. Faugeras,et al.  Variational principles, surface evolution, PDEs, level set methods, and the stereo problem , 1998, IEEE Trans. Image Process..

[12]  Adrian Hilton,et al.  Model-based multiple view reconstruction of people , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[13]  Xiaojun Wu,et al.  Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video , 2004, Comput. Vis. Image Underst..

[14]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[15]  Adrian Hilton,et al.  Virtual view synthesis of people from multiple view video sequences , 2005, Graph. Model..

[16]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[17]  Tony Ezzat,et al.  Trainable videorealistic speech animation , 2002, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[18]  Takeo Kanade,et al.  Image-based spatio-temporal modeling and view interpolation of dynamic events , 2005, TOGS.

[19]  Marcus A. Magnor,et al.  Space-time isosurface evolution for temporally coherent 3D reconstruction , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[20]  Marc Pollefeys,et al.  Multi-view reconstruction using photo-consistency and exact silhouette constraints: a maximum-flow formulation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[21]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[22]  Mark R. Stevens,et al.  Methods for Volumetric Reconstruction of Visual Scenes , 2004, International Journal of Computer Vision.

[23]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[24]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[25]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[26]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[27]  Stan Sclaroff,et al.  Stochastic refinement of the visual hull to satisfy photometric and silhouette consistency constraints , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[28]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[29]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[30]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[31]  Francis Schmitt,et al.  Silhouette and stereo fusion for 3D object modeling , 2003, Fourth International Conference on 3-D Digital Imaging and Modeling, 2003. 3DIM 2003. Proceedings..

[32]  Adrian Hilton,et al.  Video-based character animation , 2005, SCA '05.

[33]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[34]  Kiyoharu Aizawa,et al.  Motion Editing in 3D Video Database , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[35]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..