Automatic Synchronization of Background Music and Motion in Computer Animation

We synchronize background music with an animation by changing the timing of both, an approach which minimizes the damage to either. Starting from a MIDI le and motion data, feature points are extracted from both sources, paired, and then synchronized using dynamic programming to time-scale the music and to timewarp the motion. We also introduce the music graph, a directed graph which encapsulates connections between many short music sequences. By traversing a music graph we can generate large amounts of new background music, in which we expect to nd a sequence which matches the motion better than the original music.

[1]  In-Kwon Lee,et al.  Background Music Generation Using Music Texture Synthesis , 2004, ICEC.

[2]  James F. O'Brien,et al.  Synthesizing sounds from physically based motion , 2001, SIGGRAPH.

[3]  Lance Williams,et al.  Motion signal processing , 1995, SIGGRAPH.

[4]  Dinesh K. Pai,et al.  Synthesis of shape dependent sounds with physical modeling , 1996 .

[5]  Rafael Morales Bueno,et al.  Using Multiattribute Prediction Suffix Graphs to Predict and Generate Music , 2001, Computer Music Journal.

[6]  James K. Hahn,et al.  MAPPING MOTION TO SOUND AND MUSIC IN COMPUTER ANIMATION AND VE , 2007 .

[7]  Eihachiro Nakamae,et al.  Synchronizing Computer Graphics Animation and Audio , 1998, IEEE Multim..

[8]  James K. Hahn,et al.  Integrating Sounds and Motions in Virtual Environments , 1998, Presence.

[9]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, ACM Trans. Graph..

[10]  Anders Friberg,et al.  Emotional Coloring of Computer-Controlled Music Performances , 2000, Computer Music Journal.

[11]  M. Cardle,et al.  Music-driven motion editing: local motion transformations guided by music analysis , 2002, Proceedings 20th Eurographics UK Conference.

[12]  Ziv Bar-Joseph,et al.  Sound-by-numbers: motion-driven sound synthesis , 2003, SCA '03.

[13]  Yoshinori Dobashi,et al.  Real-time rendering of aerodynamic sound using sound textures based on computational fluid dynamics , 2003, ACM Trans. Graph..

[14]  Rafael Morales-Bueno,et al.  Using Multiattribute Prediction Suffix Graphs to Predict and Generate Music , 2001 .

[15]  Sung Yong Shin,et al.  Rhythmic-motion synthesis based on motion-beat analysis , 2003, ACM Trans. Graph..

[16]  Sho Yoshida,et al.  Automatic background music generation based on actors' mood and motions , 1994, Comput. Animat. Virtual Worlds.

[17]  James F. O'Brien,et al.  Synthesizing Sounds from Physically Based Motion , 2001, SIGGRAPH Video Review on Animation Theater Program.

[18]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[19]  Dinesh K. Pai,et al.  FoleyAutomatic: physically-based sound effects for interactive simulation and animation , 2001, SIGGRAPH.

[20]  Okan Arikan,et al.  Interactive motion generation from examples , 2002, ACM Trans. Graph..

[21]  Ken-ichi Anjyo,et al.  Fourier principles for emotion-based human figure animation , 1995, SIGGRAPH.

[22]  Ning Hu,et al.  A Probabilistic Model of Melodic Similarity , 2002, ICMC.

[23]  David A. Forsyth,et al.  Motion synthesis from annotations , 2003, ACM Trans. Graph..

[24]  John I. McCool,et al.  Probability and Statistics With Reliability, Queuing and Computer Science Applications , 2003, Technometrics.

[25]  Matthew Stone,et al.  Speaking with hands: creating animated conversational characters from recordings of human performance , 2004, ACM Trans. Graph..

[26]  Kishor S. Trivedi.  Probability and Statistics with Reliability, Queuing, and Computer Science Applications , 1984 .

[27]  Maja J. Mataric,et al.  Automated derivation of behavior vocabularies for autonomous humanoid motion , 2003, AAMAS '03.

[28]  Robert Rowe,et al.  Machine Musicianship , 2001 .

[29]  Maja J. Mataric,et al.  Automated Derivation of Primitives for Movement Classification , 2000, Auton. Robots.

[30]  Katsumi Tanaka,et al.  Similarity retrieval of human motion as multi-stream time series data , 1999, Proceedings 1999 International Symposium on Database Applications in Non-Traditional Environments (DANTE'99) (Cat. No.PR00496).