Conformalized Quantile Regression

Conformal prediction is a technique for constructing prediction intervals that attain valid coverage in finite samples, without making distributional assumptions. Despite this appeal, existing conformal methods can be unnecessarily conservative because they form intervals of constant or weakly varying length across the input space. In this paper we propose a new method that is fully adaptive to heteroscedasticity. It combines conformal prediction with classical quantile regression, inheriting the advantages of both. We establish a theoretical guarantee of valid coverage, supplemented by extensive experiments on popular regression datasets. We compare the efficiency of conformalized quantile regression to other conformal methods, showing that our method tends to produce shorter intervals.

[1]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[2]  Stephen Portnoy,et al.  Statistical inference on heteroscedastic models based on regression quantiles , 1998 .

[3]  James W. Taylor.  A Quantile Regression Neural Network Approach to Estimating the Conditional Density of Multiperiod Returns , 2000 .

[4]  Vladimir Vovk,et al.  Cross-conformal predictors , 2012, Annals of Mathematics and Artificial Intelligence.

[5]  Ulf Johansson,et al.  Efficient conformal regressors using bagged neural nets , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[6]  W. Gasarch,et al.  The Book Review Column 1 Coverage Untyped Systems Simple Types Recursive Types Higher-order Systems General Impression 3 Organization, and Contents of the Book , 2022 .

[7]  Henrik Boström,et al.  Regression trees for streaming data with local performance guarantees , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[8]  J. Robins,et al.  Distribution-Free Prediction Sets , 2013, Journal of the American Statistical Association.

[9]  Haris Haralambous,et al.  Reliable prediction intervals with regression neural networks , 2011, Neural Networks.

[10]  Ingo Steinwart,et al.  Estimating conditional quantiles with the help of the pinball loss , 2011, 1102.2101.

[11]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[12]  David Lopez-Paz,et al.  Frequentist uncertainty estimates for deep learning , 2018, ArXiv.

[13]  Alessandro Rinaldo,et al.  Distribution-Free Predictive Inference for Regression , 2016, Journal of the American Statistical Association.

[14]  Harris Papadopoulos,et al.  Inductive Confidence Machines for Regression , 2002, ECML.

[15]  D. Hunter,et al.  Quantile Regression via an MM Algorithm , 2000 .

[16]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[17]  Harris Papadopoulos,et al.  Normalized nonconformity measures for regression Conformal Prediction , 2008 .

[18]  Naomi S. Altman,et al.  Quantile regression , 2019, Nature Methods.

[19]  Tuve Löfström,et al.  Signed-Error Conformal Regression , 2014, PAKDD.

[20]  R. Koenker,et al.  Regression Quantiles , 2007 .

[21]  Wooseok Ha,et al.  Trimmed Conformal Prediction for High-Dimensional Models , 2016, 1611.09933.

[22]  Harris Papadopoulos,et al.  Regression Conformal Prediction with Nearest Neighbours , 2014, J. Artif. Intell. Res..

[23]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[24]  Henrik Boström,et al.  Accelerating difficulty estimation for conformal regression forests , 2017, Annals of Mathematics and Artificial Intelligence.

[25]  Alexander Gammerman,et al.  Conformal calibrators , 2019, COPA.

[26]  Alexander J. Smola,et al.  Nonparametric Quantile Estimation , 2006, J. Mach. Learn. Res..

[27]  V. Chernozhukov,et al.  QUANTILE AND PROBABILITY CURVES WITHOUT CROSSING , 2007, 0704.3649.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[30]  Larry Wasserman,et al.  Distribution‐free prediction bands for non‐parametric regression , 2014 .

[31]  A. Gammerman,et al.  On-line predictive linear regression , 2005, math/0511522.

[32]  Nicolai Meinshausen,et al.  Quantile Regression Forests , 2006, J. Mach. Learn. Res..

[33]  Dinesh Kumar,et al.  Comment Volume Prediction Using Neural Networks and Decision Trees , 2015 .

[34]  Harris Papadopoulos,et al.  Inductive Conformal Prediction: Theory and Application to Neural Networks , 2008 .

[35]  Vladimir Vovk,et al.  Nonparametric predictive distributions based on conformal prediction , 2017, Machine Learning.

[36]  Mohamed Zaki,et al.  High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach , 2018, ICML.

[37]  S. Portnoy,et al.  Direct use of regression quantiles to construct confidence sets in linear models , 1996 .

[38]  Zhigang Zeng,et al.  Landslide Displacement Prediction With Uncertainty Based on Neural Networks With Random Hidden Weights , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[39]  Roger Koenker,et al.  An empirical quantile function for linear models with iid errors , 1981 .

[40]  Rina Foygel Barber,et al.  Discretized conformal prediction for efficient distribution‐free inference , 2017, 1709.06233.

[41]  Henrik Boström,et al.  Regression conformal prediction with random forests , 2014, Machine Learning.

[42]  David Lopez-Paz,et al.  Single-Model Uncertainties for Deep Learning , 2018, NeurIPS.