Sound-by-numbers: motion-driven sound synthesis

We present the first algorithm for automatically generating soundtracks for input animation based on other animations' soundtrack. This technique can greatly simplify the production of soundtracks in computer animation and video by re-targeting existing soundtracks. A segment of source audio is used to train a statistical model which is then used to generate variants of the original audio to fit particular constraints. These constraints can either be specified explicitly by the user in the form of large-scale properties of the sound texture, or determined automatically and semi-automatically by matching similar motion events in a source animation to those in the target animation.

[1]  Richard Polfreman,et al.  Sound spotting: a frame-based approach , 2001 .

[2]  Okan Arikan,et al.  Interactive motion generation from examples , 2002, ACM Trans. Graph..

[3]  Sho Yoshida,et al.  Automatic background music generation based on actors' mood and motions , 1994, Comput. Animat. Virtual Worlds.

[4]  Dimitrios Gunopulos,et al.  Fast Motion Capture Matching with Replicated Motion Editing , 2003 .

[5]  James K. Hahn,et al.  Integrating Sounds in Virtual Environments. , 1998 .

[6]  Christoph Bregler,et al.  Motion capture assisted animation: texturing and synthesis , 2002, ACM Trans. Graph..

[7]  Tapio Takala,et al.  Sound rendering , 1992, SIGGRAPH.

[8]  Dani Lischinski,et al.  Synthesizing Sound Textures through Wavelet Tree Learning , 2002, IEEE Computer Graphics and Applications.

[9]  James F. O'Brien,et al.  Synthesizing Sounds from Physically Based Motion , 2001, SIGGRAPH Video Review on Animation Theater Program.

[10]  Dinesh K. Pai,et al.  Synthesis of shape dependent sounds with physical modeling , 1996 .

[11]  Tapio Takala,et al.  An integrated approach to motion and sound , 1995, Comput. Animat. Virtual Worlds.

[12]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[13]  Demetri Terzopoulos,et al.  Deformable models , 2000, The Visual Computer.

[14]  Dani Lischinski,et al.  Granular Synthesis of Sound Textures Using Statistical Learning , 1999, ICMC.

[15]  Lance Williams,et al.  Motion signal processing , 1995, SIGGRAPH.

[16]  Eihachiro Nakamae,et al.  Synchronizing Computer Graphics Animation and Audio , 1998, IEEE Multim..

[17]  James K. Hahn,et al.  Integrating Sounds and Motions in Virtual Environments , 1998, Presence.

[18]  Xavier Serra,et al.  Integrating complementary spectral models in the design of a musical synthesizer , 1997, ICMC.

[19]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[20]  Curtis Roads,et al.  Introduction to Granular Synthesis , 1988 .

[21]  Dinesh K. Pai,et al.  FoleyAutomatic: physically-based sound effects for interactive simulation and animation , 2001, SIGGRAPH.

[22]  Dinesh K. Pai,et al.  Manipulation and Resynthesis with Natural Grains , 2001, ICMC.

[23]  Chen Shen,et al.  Synthesizing sounds from rigid-body simulations , 2002, SCA '02.

[24]  Harry Shum,et al.  Motion texture: a two-level statistical model for character motion synthesis , 2002, ACM Trans. Graph..

[25]  Eamonn J. Keogh,et al.  Iterative Deepening Dynamic Time Warping for Time Series , 2002, SDM.

[26]  Jonathan Foote,et al.  ARTHUR: Retrieving Orchestral Music by Long-Term Structure , 2000, ISMIR.