Conformal prediction interval for dynamic time-series

We develop a method to build distribution-free prediction intervals for time-series based on conformal inference, called \Verb|EnPI| that wraps around any ensemble estimator to construct sequential prediction intervals. \Verb|EnPI| is closely related to the conformal prediction (CP) framework but does not require data exchangeability. Theoretically, these intervals attain finite-sample, approximately valid average coverage for broad classes of regression functions and time-series with strongly mixing stochastic errors. Computationally, \Verb|EnPI| requires no training of multiple ensemble estimators; it efficiently operates around an already trained ensemble estimator. In general, \Verb|EnPI| is easy to implement, scalable to producing arbitrarily many prediction intervals sequentially, and well-suited to a wide range of regression functions. We perform extensive simulations and real-data analyses to demonstrate its effectiveness.

[1]  M. Rosenblatt A CENTRAL LIMIT THEOREM AND A STRONG MIXING CONDITION. , 1956, Proceedings of the National Academy of Sciences of the United States of America.

[2]  Luc Devroye,et al.  Distribution-free performance bounds for potential function rules , 1979, IEEE Trans. Inf. Theory.

[3]  Sastry G. Pantula,et al.  A note on strong mixing of ARMA processes , 1986 .

[4]  C. Hesse Rates of convergence for the empirical distribution funciton and the empirical characteristic function of a broad class of linear processes , 1990 .

[5]  P. Doukhan Mixing: Properties and Examples , 1994 .

[6]  Naonori Ueda,et al.  Generalization error of ensemble estimators , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[7]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[8]  Xiaotong Shen,et al.  Sieve extremum estimates for weakly dependent data , 1998 .

[9]  Halbert White,et al.  Improved Rates and Asymptotic Normality for Nonparametric Neural Network Estimators , 1999, IEEE Trans. Inf. Theory.

[10]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[11]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[12]  Tomaso Poggio,et al.  Everything old is new again: a fresh look at historical approaches in machine learning , 2002 .

[13]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[14]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[15]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[16]  Harris Papadopoulos,et al.  Conformal Prediction with Neural Networks , 2007, 19th IEEE International Conference on Tools with Artificial Intelligence(ICTAI 2007).

[17]  M. Kosorok Introduction to Empirical Processes and Semiparametric Inference , 2008 .

[18]  Vladimir Vovk,et al.  A tutorial on conformal prediction , 2007, J. Mach. Learn. Res..

[19]  P. Bickel,et al.  SIMULTANEOUS ANALYSIS OF LASSO AND DANTZIG SELECTOR , 2008, 0801.1095.

[20]  Michele Modugno,et al.  Maximum Likelihood Estimation of Factor Models on Data Sets with Arbitrary Pattern of Missing Data , 2010, SSRN Electronic Journal.

[21]  S. Bobkov,et al.  Concentration of empirical distribution functions with applications to non-i.i.d. models , 2010, 1011.6165.

[22]  Efstathios Paparoditis,et al.  Bootstrap methods for dependent data: A review , 2011 .

[23]  Andreas Dengel,et al.  Histogram-based Outlier Score (HBOS): A fast Unsupervised Anomaly Detection Algorithm , 2012 .

[24]  Fei Tony Liu,et al.  Isolation-Based Anomaly Detection , 2012, TKDD.

[25]  Charu C. Aggarwal,et al.  Outlier Analysis , 2013, Springer New York.

[26]  Omar Besbes,et al.  Optimal Exploration-Exploitation in a Multi-Armed-Bandit Problem with Non-Stationary Rewards , 2014, Stochastic Systems.

[27]  D. D. Lucas,et al.  Designing optimal greenhouse gas observing networks that consider performance and cost , 2014 .

[28]  Paul Denholm,et al.  Grid Integration and the Carrying Capacity of the U.S. Grid to Incorporate Variable Renewable Energy , 2015 .

[29]  Luis M. Candanedo,et al.  Data driven prediction models of energy use of appliances in a low-energy house , 2017 .

[30]  Evgeny Burnaev,et al.  Conformal k-NN Anomaly Detector for Univariate Data Streams , 2017, COPA.

[31]  Jing He,et al.  Cautionary tales on air-quality improvement in Beijing , 2017, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[32]  Victor Chernozhukov,et al.  An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls , 2017, Journal of the American Statistical Association.

[33]  Emmanuel Rio,et al.  Asymptotic Theory of Weakly Dependent Random Processes , 2017 .

[34]  Alexander Gammerman,et al.  Inductive Conformal Martingales for Change-Point Detection , 2017, COPA.

[35]  Yishay Mansour,et al.  Discriminative Learning of Prediction Intervals , 2017, AISTATS.

[36]  Victor Chernozhukov,et al.  Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data , 2018, COLT.

[37]  Alessandro Rinaldo,et al.  Distribution-Free Predictive Inference for Regression , 2016, Journal of the American Statistical Association.

[38]  Optimal Exploration–Exploitation in a Multi-armed Bandit Problem with Non-stationary Rewards , 2019 .

[39]  Ryan J. Tibshirani,et al.  Predictive inference with the jackknife+ , 2019, The Annals of Statistics.

[40]  Yaniv Romano,et al.  Conformalized Quantile Regression , 2019, NeurIPS.

[41]  Hannes Leeb,et al.  Adaptive, Distribution-Free Prediction Intervals for Deep Neural Networks , 2019, ArXiv.

[42]  Emmanuel J. Candès,et al.  Conformal Prediction Under Covariate Shift , 2019, NeurIPS.

[43]  Yaniv Romano,et al.  Classification with Valid and Adaptive Coverage , 2020, NeurIPS.

[44]  Chen Xu,et al.  Predictive inference is free with the jackknife+-after-bootstrap , 2020, NeurIPS.

[45]  Kory D. Johnson,et al.  Adaptive, Distribution-Free Prediction Intervals for Deep Networks , 2019, AISTATS.

[46]  Gilson T. Shimizu,et al.  Distribution-free conditional predictive bands using density estimators , 2019, AISTATS.

[47]  Radu Horaud,et al.  A Comprehensive Analysis of Deep Regression , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  David Lindsay,et al.  Application of conformal prediction interval estimations to market makers' net positions , 2020, COPA.

[49]  Simone Vantini,et al.  Conformal Prediction: a Unified Review of Theory and New Challenges , 2020, ArXiv.

[50]  The limits of distribution-free conditional predictive inference , 2019, Information and Inference: A Journal of the IMA.

[51]  John C. Duchi,et al.  Robust Validation: Confident Predictions Even When Distributions Shift , 2020, ArXiv.

[52]  M. Sesia Conformal histogram regression , 2021, 2105.08747.

[53]  Michael I. Jordan,et al.  Uncertainty Sets for Image Classifiers using Conformal Prediction , 2021, ICLR.

[54]  Yao Xie,et al.  Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data , 2021, 2105.11886.