On calibration of language recognition scores

Recent publications have examined the topic of calibration of confidence scores in the field of (binary-hypothesis) speaker detection. We extend this topic to the case of multiple-hypothesis language recognition. We analyze the structure of multiple-hypothesis recognition problems to show that any such problem subsumes a multitude of derived sub-problems and that therefore the calibration of all of these problems are interrelated. We propose a simple global calibration metric that can be generally applied to a multiple-hypothesis problem and then demonstrate experimentally on some NIST-LRE-'05 data how this relates to the calibration of some of the derived binary-hypotheses sub-problems

[1]  J. Skilling,et al.  Maximum-entropy and Bayesian methods in inverse problems , 1985 .

[2]  Paola Sebastiani,et al.  Experimental design to maximise information , 2001 .

[3]  Javier Ortega-Garcia,et al.  Likelihood Ratio Calibration in a Transparent and Testable Forensic Speaker Recognition Framework , 2006, 2006 IEEE Odyssey - The Speaker and Language Recognition Workshop.

[4]  Stephen E. Fienberg,et al.  The Comparison and Evaluation of Forecasters. , 1983 .

[5]  H. P. Wynn,et al.  Experimental Design to Maximize Information , 2022 .

[6]  Rich Caruana,et al.  Predicting good probabilities with supervised learning , 2005, ICML.

[7]  C. R. Smith,et al.  Maximum-Entropy and Bayesian Methods in Inverse Problems , 1985 .

[8]  N. Dalkey.  Inductive Inference and the Maximum Entropy Principle , 1985 .

[9]  William M. Campbell,et al.  Estimating and evaluating confidence for forensic speaker recognition , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[10]  William M. Campbell,et al.  Understanding Scores in Forensic Speaker Recognition , 2006, 2006 IEEE Odyssey - The Speaker and Language Recognition Workshop.

[11]  Niko Brümmer,et al.  Application-independent evaluation of speaker detection , 2006, Comput. Speech Lang..

[12]  David A. van Leeuwen,et al.  Channel-dependent GMM and Multi-class Logistic Regression models for language recognition , 2006, Odyssey.

[13]  N. Brummer,et al.  Channel-dependent GMM and Multi-class Logistic Regression models for language recognition , 2006, 2006 IEEE Odyssey - The Speaker and Language Recognition Workshop.