A tutorial on conformal prediction

Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability e, together with a method that makes a prediction ŷ of a label y, it produces a set of labels, typically containing ŷ, that also contains y with probability 1 – e. Conformal prediction can be applied to any method for producing ŷ: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right 1 – e of the time, even though they are based on an accumulating data set rather than on independent data sets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in Algorithmic Learning in a Random World, by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).

[1]  W. R. Strong,et al.  Wahrscheinlichkeitsrechnung und ihre Anwendung auf Fehlerausgleichung Statistik und Lebensversicherung. Vol. II , 1903 .

[2]  Student,et al.  THE PROBABLE ERROR OF A MEAN , 1908 .

[3]  E. Czuber.  Wahrscheinlichkeitsrechnung, und ihre Anwendung auf Fehlerausgleichung Statistik und Lebensversicherung. Vol. I. Wahrscheinlichkeitstheorie; Fehlerausgleichung; Kollektivmasslehre , 1909 .

[4]  E. Czuber.  Wahrscheinlichkeitsrechnung, und ihre Anwendung auf Fehlerausgleichung, Statistik und Lebensversicherung , 1911 .

[5]  R. Fisher.  THE FIDUCIAL ARGUMENT IN STATISTICAL INFERENCE , 1935 .

[6]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[7]  J. Neyman.  Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability , 1937 .

[8]  L. J. Savage,et al.  Symmetric measures on Cartesian products , 1955 .

[9]  M. S. Bartlett,et al.  Statistical methods and scientific inference. , 1957 .

[10]  N. Draper,et al.  Applied Regression Analysis , 1967 .

[11]  B. Efron.  Student's t-Test under Symmetry Conditions , 1969 .

[12]  A. Dempster.  Elements of Continuous Multivariate Analysis , 1969 .

[13]  Ole E. Barndorff-Nielsen,et al.  Proceedings of Conference on Foundational Questions in Statistical Inference, Aarhus, May 7-12, 1973 , 1974 .

[14]  G. K. Robinson.  Some counterexamples to the theory of confidence intervals , 1975 .

[15]  P. Bickel,et al.  Mathematical Statistics: Basic Ideas and Selected Topics , 1977 .

[16]  S. Lauritzen.  Extremal Families and Systems of Sufficient Statistics , 1988 .

[17]  E. Asarin.  Some Properties of Kolmogorov $\Delta$-Random Finite Sequences , 1988 .

[18]  Thomas P. Ryan,et al.  Modern Regression Methods , 1996 .

[19]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[20]  Richard F. Gunst,et al.  Applied Regression Analysis , 1999, Technometrics.

[21]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[22]  G. Shafer,et al.  Probability and Finance: It's Only a Game! , 2001 .

[23]  Stephen E. Fienberg,et al.  Testing Statistical Hypotheses , 2005 .

[24]  W. Gasarch,et al.  The Book Review Column 1 Coverage Untyped Systems Simple Types Recursive Types Higher-order Systems General Impression 3 Organization, and Contents of the Book , 2022 .

[25]  Alexander Gammerman,et al.  Hedging predictions in machine learning , 2006, ArXiv.

[26]  P. Rosenbaum,et al.  R-Estimates vs. GMM: A Theoretical Case Study of Validity and Efficiency , 2006, math/0701087.

[27]  G. Shafer.  From Cournot’s Principle to Market Efficiency , 2007 .

[28]  A. Gammerman,et al.  On-line predictive linear regression , 2005, math/0511522.