What makes music musical? a framework for extracting performance expression and emotion in musical sound

We present a framework to provide a quantitative representation of aspects of musical sound that are associated with musical expressiveness and emotions. After a brief introduction to the background of expressive features in music, we introduce a score to audio mapping algorithm based on dynamic time warping, which segments the audio by comparing it to a music score. Expressive feature extraction algorithms are then introduced. The algorithms extract an expressive feature set that includes pitch deviation, loudness, timbre, timing, articulation, and modulation from the segmented audio to construct an expressive feature database. We have demonstrated these tools in the context of solo western classical music, specifically for the solo oboe. We also discuss potential applications to music performance education and music “language” processing.

[1]  Anssi Klapuri,et al.  Signal Processing Methods for Music Transcription , 2006 .

[2]  N. Geschwind.  Language and the brain. , 1972, Scientific American.

[3]  A. Christopoulos,et al.  Fitting Models to Biological Data Using Linear and Nonlinear Regression: A Practical Guide to Curve Fitting , 2004 .

[4]  W. Marsden.  I and J , 2012 .

[5]  R. Parncutt,et al.  The Science and Psychology of Music Performance Creative Strategies for Teaching and Learning , 2002 .

[6]  William Rothstein,et al.  Phrase Rhythm in Tonal Music , 1990 .

[7]  T. Moon,et al.  Mathematical Methods and Algorithms for Signal Processing , 1999 .

[8]  Meinard Müller,et al.  Information retrieval for music and motion , 2007 .

[9]  Franz Hlawatsch,et al.  Time-Frequency Analysis , 2008 .

[10]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[11]  Garry Tamlyn,et al.  Music , 1993 .

[12]  Martin Schuring.  Oboe Art and Method , 2009 .

[13]  Meinard Müller,et al.  Automatic synchronization of music data in score-, MIDI- and PCM-format , 2003, ISMIR.

[14]  D. Temperley.  Music and probability , 2006 .

[15]  Susan Tepping,et al.  Analysis of Tonal Music: A Schenkerian Approach , 1998 .

[16]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[17]  Mert Bay,et al.  Harmonic Source Separation Using Prestored Spectra , 2006, ICA.

[18]  Mark B. Sandler,et al.  A tutorial on onset detection in music signals , 2005, IEEE Transactions on Speech and Audio Processing.

[19]  N. Fletcher,et al.  Music Producers.(Book Reviews: The Physics of Musical Instruments.) , 1991 .

[20]  Mark F. Bocko,et al.  Polyphonic music transcription employing max-margin classification of spectrograhic features , 2009, 2009 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.