On-line confidence machines are well-calibrated

Transductive Confidence Machine (TCM) and its computationally efficient modification, inductive confidence machine (ICM), are ways of complementing machine-learning algorithms with practically useful measures of confidence. We show that when TCM and ICM are used in the on-line mode, their confidence measures are well-calibrated, in the sense that predictive regions at confidence level 1-/spl delta/ will be wrong with relative frequency at most /spl delta/ (approaching /spl delta/ in the case of randomised TCM and ICM) in the long run. This is not just an asymptotic phenomenon: actually the error probability of randomised TCM and ICM is d at every trial and errors happen independently at different trials.

[1]  Alexander Gammerman,et al.  Pattern Recognition and Density Estimation under the General i.i.d. Assumption , 2001, COLT/EuroCOLT.

[2]  Vladimir Vovk,et al.  Comparing the Bayes and Typicalness Frameworks , 2001, ECML.

[3]  Alexander Gammerman,et al.  Testing Exchangeability On-Line , 2003, ICML.

[4]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[5]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[6]  M. Schervish.  Theory of Statistics , 1995 .

[7]  Vladimir Vovk,et al.  Online Region Prediction with Real Teachers , 2003 .

[8]  R. K. Shyamasundar,et al.  Introduction to algorithms , 1996 .

[9]  Harris Papadopoulos,et al.  Qualified Prediction for Large Data Sets in the Case of Pattern Recognition , 2002, International Conference on Machine Learning and Applications.

[10]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[11]  Alexander Gammerman,et al.  Transductive Confidence Machines for Pattern Recognition , 2002, ECML.

[12]  Vladimir Vovk,et al.  Ridge Regression Confidence Machine , 2001, International Conference on Machine Learning.

[13]  Vladimir Vovk,et al.  A Universal Well-Calibrated Algorithm for On-line Classification , 2004, J. Mach. Learn. Res..

[14]  Alexander Gammerman,et al.  Transduction with Confidence and Credibility , 1999, IJCAI.

[15]  Vladimir Vovk,et al.  Asymptotic Optimality of Transductive Confidence Machine , 2002, ALT.

[16]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.