A comparison of some conformal quantile regression methods

We compare two recently proposed methods that combine ideas from conformal inference and quantile regression to produce locally adaptive and marginally valid prediction intervals under sample exchangeability (Romano et al., 2019; Kivaranovic et al., 2019). First, we prove that these two approaches are asymptotically efficient in large samples, under some additional assumptions. Then we compare them empirically on simulated and real data. Our results demonstrate that the method in Romano et al. (2019) typically yields tighter prediction intervals in finite samples. Finally, we discuss how to tune these procedures by fixing the relative proportions of observations used for training and conformalization.

[1]  Alessandro Rinaldo,et al.  Distribution-Free Predictive Inference for Regression , 2016, Journal of the American Statistical Association.

[2]  Alan F. Murray,et al.  Confidence estimation methods for neural networks : a practical comparison , 2001, ESANN.

[3]  Harris Papadopoulos,et al.  Regression Conformal Prediction with Nearest Neighbours , 2014, J. Artif. Intell. Res..

[4]  R. Koenker,et al.  Regression Quantiles , 2007 .

[5]  G. Shafer,et al.  Algorithmic Learning in a Random World , 2005 .

[6]  Nicolai Meinshausen,et al.  Quantile Regression Forests , 2006, J. Mach. Learn. Res..

[7]  Trevor J. Hastie,et al.  Confidence intervals for random forests: the jackknife and the infinitesimal jackknife , 2013, J. Mach. Learn. Res..

[8]  Harris Papadopoulos,et al.  Conformal Prediction with Neural Networks , 2007 .

[9]  Hannes Leeb,et al.  Adaptive, Distribution-Free Prediction Intervals for Deep Neural Networks , 2019, ArXiv.

[10]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[11]  James W. Taylor A Quantile Regression Neural Network Approach to Estimating the Conditional Density of Multiperiod Returns , 2000 .

[12]  Yaniv Romano,et al.  Conformalized Quantile Regression , 2019, NeurIPS.

[13]  Harris Papadopoulos,et al.  Inductive Conformal Prediction: Theory and Application to Neural Networks , 2008 .

[14]  Harris Papadopoulos,et al.  Inductive Confidence Machines for Regression , 2002, ECML.

[15]  Harris Papadopoulos,et al.  Normalized nonconformity measures for regression Conformal Prediction , 2008 .

[16]  A. Gammerman,et al.  On-line predictive linear regression , 2005, math/0511522.