Understanding Effects of Subjectivity in Measuring Chord Estimation Accuracy

To assess the performance of an automatic chord estimation system, reference annotations are indispensable. However, owing to the complexity of music and the sometimes ambiguous harmonic structure of polyphonic music, chord annotations are inherently subjective, and as a result any derived accuracy estimates will be subjective as well. In this paper, we investigate the extent of the confounding effect of subjectivity in reference annotations. Our results show that this effect is important, and they affect different types of automatic chord estimation systems in different ways. Our results have implications for research on automatic chord estimation, but also on other fields that evaluate performance by comparing against human provided annotations that are confounded by subjectivity.

[1]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[2]  Mark B. Sandler,et al.  Symbolic Representation of Musical Chords: A Proposed Syntax for Text Annotations , 2005, ISMIR.

[3]  Christopher Harte,et al.  Towards automatic extraction of harmony information from music signals , 2010 .

[4]  Shigeki Sagayama,et al.  MIREX 2010 : JOINT RECOGNITION OF KEY AND CHORD FROM MUSIC AUDIO SIGNALS USING KEY-MODULATION HMM , 2010 .

[5]  Lawrence R. Rabiner,et al.  A tutorial on Hidden Markov Models , 1986 .

[6]  Johan Pauwels,et al.  Evaluating automatically estimated chord sequences , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[7]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[8]  Pietro Perona,et al.  Inferring Ground Truth from Subjective Labelling of Venus Images , 1994, NIPS.

[9]  Maurizio Omologo,et al.  Use of Hidden Markov Models and Factored Language Models for Automatic Chord Recognition , 2009, ISMIR.

[10]  J. Stephen Downie,et al.  The music information retrieval evaluation exchange (2005-2007): A window into music information retrieval research , 2008, Acoustical Science and Technology.

[11]  Matthias Mauch,et al.  Automatic chord transcription from audio using computational models of musical context , 2010 .

[12]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[13]  Tijl De Bie,et al.  An End-to-End Machine Learning System for Harmonic Analysis of Music , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[14]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[15]  Juan Pablo Bello,et al.  A Feature Smoothing Method for Chord Recognition Using Recurrence Plots , 2011, ISMIR.

[16]  Laurent Oudre,et al.  Concurrent Estimation of Chords and Keys from Audio , 2010, ISMIR.

[17]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[18]  Tijl De Bie,et al.  Automatic Chord Estimation from Audio: A Review of the State of the Art , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[19]  Daniel P. W. Ellis,et al.  Structured Prediction Models for Chord Transcription of Music Audio , 2009, 2009 International Conference on Machine Learning and Applications.

[20]  Geoffroy Peeters,et al.  Joint Estimation of Chords and Downbeats From an Audio Signal , 2011, IEEE Transactions on Audio, Speech, and Language Processing.