Camera Motion Style Transfer

When depicting both virtual and physical worlds, the viewer's impression of presence in these worlds is strongly linked to camera motion. Plausible and artist-controlled camera movement can substantially increase scene immersion. While physical camera motion exhibits subtle details of position, rotation, and acceleration, these details are often missing for virtual camera motion. In this work, we analyze camera movement using signal theory. Our system allows us to stylize a smooth user-defined virtual base camera motion by enriching it with plausible details. A key component of our system is a database of videos filmed by physical cameras. These videos are analyzed with a camera-motion estimation algorithm (structure-from-motion) and labeled manually with a specific style. By considering spectral properties of location, orientation and acceleration, our solution learns camera motion details. Consequently, an arbitrary virtual base motion, defined in any conventional animation package, can be automatically modified according to a user-selected style. As shown in our experiments, the resulting shots are still fully artist-controlled, but appear richer and more physically plausible.

[1]  Lance Williams,et al.  Motion signal processing , 1995, SIGGRAPH.

[2]  J. Bennett.  Vision and Art: The Biology of Seeing , 2003 .

[3]  Rama Chellappa,et al.  Evaluation of image stabilization algorithms , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[4]  Sylvain Lefebvre,et al.  State of the Art in Example-based Texture Synthesis , 2009, Eurographics.

[5]  Michael Gleicher,et al.  Re-cinematography: improving the camera dynamics of casual video , 2007, ACM Multimedia.

[6]  Frédo Durand,et al.  Two-scale tone management for photographic look , 2006, SIGGRAPH 2006.

[7]  Wojciech Matusik,et al.  A statistical model for synthesis of detailed facial geometry , 2006, SIGGRAPH 2006.

[8]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[9]  Steven M. Drucker,et al.  CINEMA: a system for procedural camera movements , 1992, I3D '92.

[10]  Donald S. Fussell,et al.  Computer rendering of stochastic models , 1998 .

[11]  Michael Gleicher,et al.  Through-the-lens camera control , 1992, SIGGRAPH.

[12]  Gabriel Taubin,et al.  A signal processing approach to fair surface design , 1995, SIGGRAPH.

[13]  James F. Blinn,et al.  Where am I? What am I looking at? (cinematography) , 1988, IEEE Computer Graphics and Applications.

[14]  Toby Howard,et al.  Accurate camera calibration for off-line, video-based augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[15]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[16]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[17]  Satoru Kawai,et al.  A simple method for computing general position in displaying three-dimensional objects , 1988, Comput. Vis. Graph. Image Process..

[18]  Reinhard Koch,et al.  Visual Modeling with a Hand-Held Camera , 2004, International Journal of Computer Vision.

[19]  Lutz Bacher.  Max Ophuls in the Hollywood Studios , 1996 .

[20]  C. Karen Liu,et al.  Learning physics-based motion style with nonlinear inverse optimization , 2005, ACM Trans. Graph..

[21]  Ken Perlin,et al.  [Computer Graphics]: Three-Dimensional Graphics and Realism , 2022 .

[22]  Alexei A. Efros,et al.  Texture synthesis by non-parametric sampling , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[23]  S. B. Kang,et al.  Image deblurring using inertial measurement sensors , 2010, SIGGRAPH 2010.

[24]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[25]  Shree K. Nayar,et al.  Priors for Large Photo Collections and What They Reveal about Cameras , 2008, ECCV.

[26]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[27]  Maneesh Agrawala,et al.  The cartoon animation filter , 2006, ACM Trans. Graph..

[28]  David Salesin,et al.  The virtual cinematographer: a paradigm for automatic real-time camera control and directing , 1996, SIGGRAPH.

[29]  Michael Gleicher,et al.  Retargetting motion to new characters , 1998, SIGGRAPH.

[30]  Jiawen Chen,et al.  Texture transfer using geometry correlation , 2006, EGSR '06.

[31]  Ares Lagae,et al.  State of the Art in Procedural Noise Functions , 2010, Eurographics.

[32]  Aaron Hertzmann,et al.  Style machines , 2000, SIGGRAPH 2000.

[33]  Michael Gleicher,et al.  Content-preserving warps for 3D video stabilization , 2009, ACM Trans. Graph..

[34]  Marc Davis,et al.  IDIC: assembling video sequences from story plans and content annotations , 1994, 1994 Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[35]  Ken-ichi Anjyo,et al.  Animating Pictures of Fluid using Video Examples , 2009, Comput. Graph. Forum.