Model Selection Using Efficiency of Conformal Predictors

The Conformal Prediction framework guarantees error calibration in the online setting, but its practical usefulness in real-world problems is affected by its efficiency, i.e. the size of the prediction region. Narrow prediction regions that maintain validity would be the most useful conformal predictors. In this work, we use the efficiency of conformal predictors as a measure to perform model selection in classifiers. We pose this objective as an optimization problem on the model parameters, and test this approach with the k-Nearest Neighbour classifier. Our results on the USPS and other standard datasets show promise in this approach.

[1]  Sethuraman Panchanathan,et al.  Kernel Learning for Efficiency Maximization in the Conformal Predictions Framework , 2010, 2010 Ninth International Conference on Machine Learning and Applications.

[2]  Vladimir Vovk,et al.  On-line confidence machines are well-calibrated , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[3]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[4]  Vladimir Vovk,et al.  Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications , 2014 .

[5]  Alexander Gammerman,et al.  Criteria of Efficiency for Conformal Prediction , 2016, COPA.

[6]  John Shawe-Taylor,et al.  A Nonconformity Approach to Model Selection for SVMs , 2009 .

[7]  Fan Yang,et al.  Distance Metric Learning-Based Conformal Predictor , 2012, AIAI.

[8]  G. Shafer,et al.  Algorithmic Learning in a Random World , 2005 .

[9]  Ashley J. Llorens,et al.  Local distance metric learning for efficient conformal predictors , 2012, 2012 IEEE International Workshop on Machine Learning for Signal Processing.