Conditional validity of inductive conformal predictors

Conformal predictors are set predictors that are automatically valid in the sense of having coverage probability equal to or exceeding a given confidence level. Inductive conformal predictors are a computationally efficient version of conformal predictors satisfying the same property of validity. However, inductive conformal predictors have only been known to control unconditional coverage probability. This paper explores various versions of conditional validity and various ways to achieve them using inductive conformal predictors and their modifications. In particular, it discusses a convenient expression of one of the modifications in terms of ROC curves.

[1]  James M. Robins,et al.  Efficient Nonparametric Conformal Prediction Regions , 2011, ArXiv.

[2]  J. Langford.  Tutorial on Practical Prediction Theory for Classification , 2005, J. Mach. Learn. Res..

[3]  Harris Papadopoulos,et al.  Inductive Confidence Machines for Regression , 2002, ECML.

[4]  J. K. Ord,et al.  Statistical Tolerance Regions: Classical and Bayesian , 1971 .

[5]  Alexander Gammerman,et al.  Transduction with Confidence and Credibility , 1999, IJCAI.

[6]  J. Tukey,et al.  Non-Parametric Estimation. I. Validation of Order Statistics , 1945 .

[7]  Alexandre B. Tsybakov,et al.  Introduction to Nonparametric Estimation , 2008, Springer series in statistics.

[8]  W. Gasarch,et al.  The Book Review Column 1 Coverage Untyped Systems Simple Types Recursive Types Higher-order Systems General Impression 3 Organization, and Contents of the Book , 2022 .

[9]  Larry A. Wasserman,et al.  Distribution Free Prediction Bands , 2012, ArXiv.

[10]  J. Friedman.  Stochastic gradient boosting , 2002 .

[11]  Ida G. Sprinkhuizen-Kuyper,et al.  A Comparison of Two Approaches to Classify with Guaranteed Performance , 2007, PKDD.

[12]  Ilia Nouretdinov.  Offline Nearest Neighbour Transductive Confidence Machine , 2008, Industrial Conference on Data Mining - Posters and Workshops.

[13]  J. Robins,et al.  Distribution-Free Prediction Sets , 2013, Journal of the American Statistical Association.

[14]  S. S. Wilks.  Determination of Sample Sizes for Setting Tolerance Limits , 1941 .

[15]  Alexander Gammerman,et al.  Conditional Prediction Intervals for Linear Regression , 2009, 2009 International Conference on Machine Learning and Applications.

[16]  Vladimir Vovk,et al.  On-line confidence machines are well-calibrated , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[17]  Samy Bengio,et al.  The Expected Performance Curve , 2003, ICML 2003.

[18]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[19]  Donald Fraser,et al.  Nonparametric Estimation IV , 1951 .

[20]  Raghu Kacker,et al.  Digital Library of Mathematical Functions , 2003 .

[21]  Berton H. Gunter,et al.  Data Analysis and Graphics Using R: An Example-Based Approach , 2004, Technometrics.

[22]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[23]  John W. Tukey,et al.  Nonparametric Estimation, III. Statistically Equivalent Blocks and Multivariate Tolerance Regions--The Discontinuous Case , 1948 .

[24]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[25]  Harris Papadopoulos,et al.  Qualified Prediction for Large Data Sets in the Case of Pattern Recognition , 2002, International Conference on Machine Learning and Applications.

[26]  Susan Starkings,et al.  Data Analysis and Graphics Using R: An Example‐based Approach, 2nd Edition by John Maindonald, John Braun , 2007 .

[27]  J. Tukey.  Non-Parametric Estimation II. Statistically Equivalent Blocks and Tolerance Regions--The Continuous Case , 1947 .

[28]  D. Fraser.  Nonparametric methods in statistics , 1957 .

[29]  Laurens van der Maaten,et al.  Off-Line Learning with Transductive Confidence Machines: An Empirical Evaluation , 2007, MLDM.

[30]  Vladimir Vovk,et al.  Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications , 2014 .

[31]  Larry Wasserman,et al.  Distribution‐free prediction bands for non‐parametric regression , 2014 .

[32]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .