Human motion synthesis from 3D video

Multiple view 3D video reconstruction of actor performance captures a level-of-detail for body and clothing movement which is time-consuming to produce using existing animation tools. In this paper we present a framework for concatenative synthesis from multiple 3D video sequences according to user constraints on movement, position and timing. Multiple 3D video sequences of an actor performing different movements are automatically constructed into a surface motion graph which represents the possible transitions with similar shape and motion between sequences without unnatural movement artifacts. Shape similarity over an adaptive temporal window is used to identify transitions between 3D video sequences. Novel 3D video sequences are synthesized by finding the optimal path in the surface motion graph between user specified key-frames for control of movement, location and timing. The optimal path which satisfies the user constraints whilst minimizing the total transition cost between 3D video sequences is found using integer linear programming. Results demonstrate that this framework allows flexible production of novel 3D video sequences which preserve the detailed dynamics of the captured movement for an actress with loose clothing and long hair without visible artifacts.

[1]  Hans Peter Graf,et al.  Sample-based synthesis of photo-realistic talking heads , 1998, Proceedings Computer Animation '98 (Cat. No.98EX169).

[2]  Nancy S. Pollard,et al.  Evaluating motion graphs for character animation , 2007, TOGS.

[3]  Okan Arikan,et al.  Interactive motion generation from examples , 2002, ACM Trans. Graph..

[4]  Adrian Hilton,et al.  Spherical matching for temporal correspondence of non-rigid surfaces , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[5]  Adrian Hilton,et al.  Temporal 3D shape matching , 2007 .

[6]  Hans-Peter Kriegel,et al.  3D Shape Histograms for Similarity Search and Classification in Spatial Databases , 1999, SSD.

[7]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[8]  Hans-Peter Seidel,et al.  Seeing People in Different Light — Joint Shape , Motion , and Reflectance Capture , 2007 .

[9]  Adrian Hilton,et al.  Correspondence labelling for wide-timeframe free-form surface matching , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[10]  Christoph Bregler,et al.  Video Rewrite: Driving Visual Speech with Audio , 1997, SIGGRAPH.

[11]  Adrian Hilton,et al.  A Study of Shape Similarity for Temporal Surface Sequences of People , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[12]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH Classes.

[13]  Szymon Rusinkiewicz,et al.  Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors , 2003, Symposium on Geometry Processing.

[14]  Harry Shum,et al.  Motion texture: a two-level statistical model for character motion synthesis , 2002, ACM Trans. Graph..

[15]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, ACM Trans. Graph..

[16]  Irfan A. Essa,et al.  Controlled animation of video sprites , 2002, SCA '02.

[17]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[18]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[19]  Adrian Hilton,et al.  Realistic synthesis of novel human movements from a database of motion capture examples , 2000, Proceedings Workshop on Human Motion.

[20]  Adrian Hilton,et al.  Video-based character animation , 2005, SCA '05.

[21]  Bobby Bodenheimer,et al.  Synthesis and evaluation of linear motion transitions , 2008, TOGS.

[22]  Andrew E. Johnson,et al.  Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Christian Rössl,et al.  Dense correspondence finding for parametrization-free animation reconstruction from video , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Bernard Chazelle,et al.  Shape distributions , 2002, TOGS.

[25]  Kiyoharu Aizawa,et al.  Motion Editing in 3D Video Database , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[26]  Radu Horaud,et al.  Temporal Surface Tracking Using Mesh Evolution , 2008, ECCV.