Comparison of subjective and objective evaluation methods for audio source separation

3937 The evaluation of audio separation algorithms can either be performed objectively by calculation of numerical measures, or subjectively through listening tests. Although objective evaluation is inherently more straightforward, subjective listening tests are still essential in determining the perceived quality of separation. This paper aims to find relationships between objective and subjective results so that numerical values can be translated into perceptual criteria. A generic audio source separation system was modelled which provided varying levels of interference, noise and artifacts. This enabled a full spread of objective measurement values to be obtained. Extensive tests were performed utilising the output synthesised by this separation model. The relationships found were presented and the factors of prime importance were determined.

[1]  H. Sabine.  Room Acoustics , 1953, The SAGE Encyclopedia of Human Communication Sciences and Disorders.

[2]  Ahmet M. Kondoz,et al.  Digital Speech: Coding for Low Bit Rate Communication Systems , 1995 .

[3]  Paris Smaragdis,et al.  Information theoretic approaches to source separation , 1997 .

[4]  L. Parra,et al.  Convolutive blind source separation based on multiple decorrelation , 1998, Neural Networks for Signal Processing VIII. Proceedings of the 1998 IEEE Signal Processing Society Workshop (Cat. No.98TH8378).

[5]  Methods for the subjective assessment of small impairments in audio systems , 2015 .

[6]  Paris Smaragdis,et al.  Blind separation of convolved mixtures in the frequency domain , 1998, Neurocomputing.

[7]  Paris Smaragdis,et al.  Evaluation of blind signal separation methods , 1999 .

[8]  George K. Kokkinakis,et al.  Blind speech separation of moving speakers in real reverberant environments , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[9]  Rémi Gribonval,et al.  BSS_EVAL Toolbox User Guide -- Revision 2.0 , 2005 .

[10]  Rémi Gribonval,et al.  Performance measurement in blind audio source separation , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[11]  Emmanuel Vincent,et al.  Preliminary guidelines for subjective evalutation of audio source separation algorithms , 2006 .