Transductive Reliability Estimation for Individual Classifications in Machine Learning and Data Mining

Machine learning and data mining approaches are nowadays being used in many fields as valuable data analysis tools. However, their serious practical use is affected by the fact, that more often than not, they cannot produce reliable and unbiased assessments of their predictions’ quality. In last years, several approaches for estimating reliability or confidence of individual classifiers have emerged, many of them building upon the algorithmic theory of randomness, such as (historically ordered) transduction-based confidence estimation, typicalness-based confidence estimation, and transductive reliability estimation. In the chapter we describe typicalness and transductive reliability estimation frameworks and propose a joint approach that compensates their weaknesses by integrating typicalness-based confidence estimation and transductive reliability estimation into a joint confidence machine. The resulting confidence machine produces confidence values in the statistical sense (e.g., a confidence level of 95% means that in 95% the predicted class is also a true class), as well as provides us with a general principle that is independent of to the particular underlying classifier.

[1]  Vladimir Vovk,et al.  Comparing the Bayes and Typicalness Frameworks , 2001, ECML.

[2]  Pat Langley,et al.  Estimating Continuous Distributions in Bayesian Classifiers , 1995, UAI.

[3]  Andreas S. Weigend,et al.  Predictions with Confidence Intervals ( Local Error Bars ) , 1994 .

[4]  Matjaz Kukar,et al.  Transductive reliability estimation for medical diagnosis , 2003, Artif. Intell. Medicine.

[5]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[6]  Avrim Blum,et al.  The Bottleneck , 2021, Monopsony Capitalism.

[7]  Marko Robnik-Sikonja,et al.  Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF , 2004, Applied Intelligence.

[8]  Alexander Gammerman,et al.  Learning by Transduction , 1998, UAI.

[9]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[10]  Inder Jeet Taneja,et al.  On Generalized Information Measures and Their Applications , 1989 .

[11]  Alison L Gibbs,et al.  On Choosing and Bounding Probability Metrics , 2002, math/0209021.

[12]  Gunnar Rätsch,et al.  Learning to Predict the Leave-One-Out Error of Kernel Based Classifiers , 2001, ICANN.

[13]  Alexander G. Gray,et al.  Retrofitting Decision Tree Classifiers Using Kernel Density Estimation , 1995, ICML.

[14]  Brian D. Ripley,et al.  Modern Applied Statistics with S Fourth edition , 2002 .

[15]  Alexander Gammerman,et al.  Transduction with Confidence and Credibility , 1999, IJCAI.

[16]  Jack P. C. Kleijnen.  Experimental Design for Sensitivity Analysis of Simulation Models , 2001 .

[17]  Dale Schuurmans,et al.  Data perturbation for escaping local maxima in learning , 2002, AAAI/IAAI.

[18]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[19]  Stefano Tarantola,et al.  Introduction to Sensitivity Analysis , 2008 .

[20]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[21]  Hilan Bensusan,et al.  Meta-Learning by Landmarking Various Learning Algorithms , 2000, ICML.

[22]  Padraig Cunningham,et al.  Confidence and prediction intervals for neural network ensembles , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[23]  Pat Langley,et al.  Editorial: On Machine Learning , 1986, Machine Learning.

[24]  Igor Kononenko,et al.  Semi-Naive Bayesian Classifier , 1991, EWSL.

[25]  Alexander Gammerman,et al.  Computationally Efficient Transductive Machines , 2000, ALT.

[26]  Tom Heskes,et al.  Practical Confidence and Prediction Intervals , 1996, NIPS.

[27]  Johannes Fürnkranz,et al.  An Evaluation of Grading Classifiers , 2001, IDA.

[28]  Igor Kononenko,et al.  Reliable Classifications with Machine Learning , 2002, ECML.

[29]  G. Diamond,et al.  Analysis of probability as an aid in the clinical diagnosis of coronary-artery disease. , 1979, The New England journal of medicine.

[30]  Dana Ron,et al.  Algorithmic Stability and Sanity-Check Bounds for Leave-One-Out Cross-Validation , 1997, Neural Computation.

[31]  M. Olona-Cabases,et al.  The probability of a correct diagnosis , 1994 .

[32]  Harry Wechsler,et al.  Transductive confidence machine for active learning , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[33]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[34]  Alexander Gammerman,et al.  Transductive Confidence Machines for Pattern Recognition , 2002, ECML.

[35]  Fabio Roli,et al.  Dynamic classifier selection based on multiple classifier behaviour , 2001, Pattern Recognit..

[36]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.

[37]  Vladimir Vovk,et al.  Ridge Regression Confidence Machine , 2001, International Conference on Machine Learning.

[38]  Mauro Birattari,et al.  Local learning for data analysis , 1998 .

[39]  William N. Venables,et al.  Modern Applied Statistics with S-Plus. , 1996 .

[40]  Matjaz Kukar,et al.  Quality assessment of individual classifications in machine learning and data mining , 2006, Knowledge and Information Systems.

[41]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[42]  O. M. Halck,et al.  Using Hard Classifiers to Estimate Conditional Class Probabilities , 2002, ECML.

[43]  Igor Kononenko,et al.  Estimation of individual prediction reliability using the local sensitivity analysis , 2008, Applied Intelligence.