PopMash: an automatic musical-mashup system using computation of musical and lyrical agreement for transitions

Musical-mashup is a popular form of music re-creation, aiming at combining multiple pieces of music to create new music artworks. Presently, it is also a challenge in the field of music information study. In this work, an effective framework for harmonious musical-mashup generation is provided. In the experiment, lyrics, melody, and rhythm of music were synthetically analyzed. The “harmony” of mashup transition was evaluated in view of the similarity scores of rhythm, melody and lyrics rhyme. “Mashupable” song segments were selected based on the transition harmony evaluation. Then, the musical-mashup output was carried out by adjusting the rhythm, tone and loudness of each segment. Finally, we created PopMash based on the proposed method, an automatic musical-mashup system that can make smooth and harmony transitions from multiple perspectives, which can efficiently reduce the manual work of music recreation.

[1]  Jyh-Shing Roger Jang,et al.  Bridging Music via Sound Effects , 2014, 2014 IEEE International Symposium on Multimedia.

[2]  Carlo Fischione,et al.  Internet of Musical Things: Vision and Challenges , 2018, IEEE Access.

[3]  P. Ivanov,et al.  Variance fluctuations in nonstationary time series: a comparative study of music genres , 2003, cond-mat/0312380.

[4]  Donncha O Maidín.  A geometrical algorithm for melodic difference , 1998 .

[5]  Wei Tsang Ooi,et al.  MoViMash: online mobile video mashup , 2012, ACM Multimedia.

[6]  M. Ross,et al.  Average magnitude difference function pitch extractor , 1974 .

[7]  Wei Tsang Ooi,et al.  Jiku director: a mobile video mashup system , 2013, MM '13.

[8]  Ja-Ling Wu,et al.  Music Cut and Paste: A Personalized Musical Medley Generating System , 2013, ISMIR.

[9]  B. Snyder.  Music and Memory: An Introduction , 2001 .

[10]  Mohan S. Kankanhalli,et al.  Content-based music structure analysis with applications to music semantics understanding , 2004, MULTIMEDIA '04.

[11]  Quan Quan,et al.  A multi-phase blending method with incremental intensity for training detection networks , 2020, The Visual Computer.

[12]  Norbert Diehl,et al.  Object-oriented motion estimation and segmentation in image sequences , 1991, Signal Process. Image Commun..

[13]  Sumit Basu,et al.  Mixing with Mozart , 2004, ICMC.

[14]  Keiichiro Hoashi,et al.  Full-Automatic DJ Mixing System with Optimal Tempo Adjustment based on Measurement Function of User Discomfort , 2009, ISMIR.

[15]  Dirk Moelants,et al.  Extracting the perceptual tempo from music , 2004, ISMIR.

[16]  Mukesh Kumar Saini,et al.  The One Man Show , 2017, 2017 IEEE International Symposium on Multimedia (ISM).

[17]  Peter H. N. de With,et al.  Automatic mashup generation from multiple-camera concert recordings , 2010, ACM Multimedia.

[18]  J. Stephen Downie,et al.  Evaluation of a simple and effective music information retrieval method , 2000, SIGIR '00.

[19]  Marsha Berman,et al.  The directory of tunes and musical themes , 1975 .

[20]  Shigeo Morishima,et al.  Automatic Mash Up Music Video Generation System by Remixing Existing Video Content , 2013, 2013 International Conference on Culture and Computing.

[21]  Eric Moulines,et al.  Non-parametric techniques for pitch-scale and time-scale modification of speech , 1995, Speech Commun..

[22]  Kyogu Lee,et al.  Lyrics-to-Audio Alignment by Unsupervised Discovery of Repetitive Patterns in Vowel Acoustics , 2017, IEEE Access.

[23]  Ja-Ling Wu,et al.  Music Paste: Concatenating Music Clips based on Chroma and Rhythm Features , 2009, ISMIR.

[24]  Matthew E. P. Davies,et al.  Psychoacoustic Approaches for Harmonic Music Mixing , 2016 .

[25]  Masataka Goto,et al.  A chorus-section detecting method for musical audio signals , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[26]  Craig G. Nevill-Manning,et al.  Distance metrics and indexing strategies for a digital library of popular music , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[27]  Yiteng Pan,et al.  A correlative denoising autoencoder to model social influence for top-N recommender system , 2019, Frontiers of Computer Science.

[28]  Youngmoo E. Kim,et al.  Analysis of a Contour-based Representation for Melody , 2000, ISMIR.

[29]  Yilin Chen,et al.  A new haze removal approach for sky/river alike scenes based on external and internal clues , 2019, Multimedia Tools and Applications.

[30]  Gilberto Bernardes,et al.  MixMash: A Visualisation System for Musical Mashup Creation , 2018, 2018 22nd International Conference Information Visualisation (IV).

[31]  Fazhi He,et al.  DRCDN: learning deep residual convolutional dehazing networks , 2019, The Visual Computer.

[32]  L. Hofmann-Engl.  RHYTHMIC SIMILARITY: A THEORETICAL AND EMPIRICAL APPROACH , 2002 .

[33]  Yiteng Pan,et al.  A scalable region-based level set method using adaptive bilateral filter for noisy image segmentation , 2019, Multimedia Tools and Applications.

[34]  Matthew E. P. Davies,et al.  AutoMashUpper: Automatic Creation of Multi-Song Music Mashups , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[35]  DeLiang Wang,et al.  Detecting pitch of singing voice in polyphonic audio , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[36]  Youngmoo E. Kim,et al.  Beat-Sync-Mash-Coder: A web application for real-time creation of beat-synchronous music mashups , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[37]  Mark B. Sandler,et al.  Structural Segmentation of Musical Audio by Constrained Clustering , 2008, IEEE Transactions on Audio, Speech, and Language Processing.

[38]  Masataka Goto,et al.  Grand Challenges in Music Information Research , 2012, Multimodal Music Processing.

[39]  Simon Langford.  The remix manual , 2011 .