Conformalized Kernel Ridge Regression

General predictive models do not provide a measure of confidence in predictions without Bayesian assumptions. A way to circumvent potential restrictions is to use conformal methods for constructing non-parametric confidence regions, that offer guarantees regarding validity. In this paper we provide a detailed description of a computationally efficient conformal procedure for Kernel Ridge Regression (KRR), and conduct a comparative numerical study to see how well conformal regions perform against the Bayesian confidence sets. The results suggest that conformalized KRR can yield predictive confidence regions with specified coverage rate, which is essential in constructing anomaly detection systems based on predictive models.

[1]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[2]  Heiko Hoffmann,et al.  Kernel PCA for novelty detection , 2007, Pattern Recognit..

[3]  Evgeny Burnaev,et al.  Model selection for anomaly detection , 2015, International Conference on Machine Vision.

[4]  Vladimir Vovk,et al.  Efficiency of conformalized ridge regression , 2014, COLT.

[5]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[6]  Philip S. Yu,et al.  Outlier Detection with Uncertain Data , 2008, SDM.

[7]  Alexander P. Kuleshov,et al.  Extended Regression on Manifolds Estimation , 2016, COPA.

[8]  Evgeny Burnaev,et al.  Computationally efficient algorithm for Gaussian Process regression in case of structured samples , 2016, Computational Mathematics and Mathematical Physics.

[9]  W. Gasarch,et al.  The Book Review Column 1 Coverage Untyped Systems Simple Types Recursive Types Higher-order Systems General Impression 3 Organization, and Contents of the Book , 2022 .

[10]  M. F. Augusteijn,et al.  Neural network classification and novelty detection , 2002 .

[11]  Pasi Fränti,et al.  Outlier Detection Using k-Nearest Neighbour Graph , 2004, ICPR.

[12]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[13]  Evgeny Burnaev,et al.  Gaussian Process Regression for Structured Data Sets , 2015, SLDS.

[14]  Alexander P. Kuleshov,et al.  Manifold Learning in Regression Tasks , 2015, SLDS.

[15]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[16]  D. W. Scott.  Kernel Density Estimators , 2008 .

[17]  Evgeny Burnaev,et al.  The Bernstein-von Mises theorem for regression based on Gaussian Processes , 2013 .

[18]  Evgeny V. Burnaev,et al.  Properties of the posterior distribution of a regression model based on Gaussian random fields , 2013, Autom. Remote. Control..

[19]  Evgeny Burnaev,et al.  Nonparametric decomposition of quasi-periodic time series for change-point detection , 2015, International Conference on Machine Vision.

[20]  Pavel Erofeev,et al.  Application of Rare Event Anticipation Techniques to Aircraft Health Management , 2014 .

[21]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[22]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[23]  Hongxing He,et al.  Outlier Detection Using Replicator Neural Networks , 2002, DaWaK.

[24]  Hans-Peter Kriegel,et al.  LoOP: local outlier probabilities , 2009, CIKM.

[25]  Maxim Panov,et al.  Regression on the basis of nonstationary Gaussian processes with Bayesian regularization , 2016 .