Image-space modal bases for plausible manipulation of objects in video

We present algorithms for extracting an image-space representation of object structure from video and using it to synthesize physically plausible animations of objects responding to new, previously unseen forces. Our representation of structure is derived from an image-space analysis of modal object deformation: projections of an object's resonant modes are recovered from the temporal spectra of optical flow in a video, and used as a basis for the image-space simulation of object dynamics. We describe how to extract this basis from video, and show that it can be used to create physically-plausible animations of objects without any knowledge of scene geometry or material properties.

[1]  Peter Avitabile,et al.  3D digital image correlation methods for full-field vibration measurement , 2011 .

[2]  Hujun Bao,et al.  Space-time editing of elastic motion through material optimization and reduction , 2014, ACM Trans. Graph..

[3]  Frédo Durand,et al.  Modal identification of simple structures with high-speed video using motion magnification , 2015 .

[4]  Dinesh K. Pai,et al.  DyRT: dynamic response textures for real time deformation simulation with graphics hardware , 2002, SIGGRAPH.

[5]  Oral Büyüköztürk,et al.  Operational and defect parameters concerning the acoustic-laser vibrometry method for FRP-reinforced concrete , 2015 .

[6]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[7]  Rune Brincker,et al.  Why Output-Only Modal Testing is a Desirable Tool for a Wide Range of Practical Applications , 2002 .

[8]  Abhijit Sarkar,et al.  Theory of Vibration , 2010 .

[9]  Jos Stam,et al.  Stochastic Dynamics: Simulating the Effects of Turbulence on Flexible Structures , 1997 .

[10]  Alex Pentland,et al.  Good vibrations: modal dynamics for graphics and animation , 1989, SIGGRAPH.

[11]  Edward H. Adelson,et al.  Shiftable multiscale transforms , 1992, IEEE Trans. Inf. Theory.

[12]  Frédo Durand,et al.  Visual vibrometry: Estimating material properties from small motions in video , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  T.,et al.  Shiftable Multi-scale TransformsEero , 1992 .

[14]  Frédo Durand,et al.  Visual vibrometry: Estimating material properties from small motions in video , 2015, CVPR.

[15]  Stefano Soatto,et al.  Dynamic Textures , 2003, International Journal of Computer Vision.

[16]  Frédo Durand,et al.  Phase-based video motion processing , 2013, ACM Trans. Graph..

[17]  Carlos E. Ventura,et al.  Why Output Only Modal Analysis is a Desirable Tool for a Wide Range of Practical Applications , 2003 .

[18]  Meng Sun,et al.  Video input driven animation (VIDA) , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[19]  Thomas S. Huang,et al.  Connected vibrations: a modal analysis approach for non-rigid motion tracking , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[20]  Dinesh K. Pai,et al.  Multiresolution green's function methods for interactive simulation of large-scale elastostatic objects , 2003, TOGS.

[21]  David Salesin,et al.  Animating pictures with stochastic motion textures , 2005, ACM Trans. Graph..

[22]  Kun Zhou,et al.  Interactive Shape Interpolation through Controllable Dynamic Deformation , 2011, IEEE Transactions on Visualization and Computer Graphics.

[23]  Frédo Durand,et al.  Riesz pyramids for fast phase-based video magnification , 2014, 2014 IEEE International Conference on Computational Photography (ICCP).

[24]  Bart Peeters,et al.  Benchmark study on system identification through ambient vibration measurements , 2000 .

[25]  Martin Szummer,et al.  Temporal texture modeling , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[26]  DurandFrédo,et al.  Image-space modal bases for plausible manipulation of objects in video , 2015 .

[27]  Alex Pentland,et al.  Closed-form solutions for physically-based shape modeling and recognition , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[28]  K. Bathe.  Finite Element Procedures , 1995 .

[29]  A. Shabana.  Theory of vibration , 1991 .

[30]  Frédo Durand,et al.  The visual microphone , 2014, ACM Trans. Graph..