High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach

This paper considers the generation of prediction intervals (PIs) by neural networks for quantifying uncertainty in regression tasks. It is axiomatic that high-quality PIs should be as narrow as possible, whilst capturing a specified portion of data. We derive a loss function directly from this axiom that requires no distributional assumption. We show how its form derives from a likelihood principle, that it can be used with gradient descent, and that model uncertainty is accounted for in ensembled form. Benchmark experiments show the method outperforms current state-of-the-art uncertainty quantification methods, reducing average PI width by over 10%.

[1]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[2]  Jürgen Schmidhuber,et al.  Characteristic Kernels on Structured Domains Excel in Robotics and Human Action Recognition , 2010, ECML/PKDD.

[3]  Vladimir Vovk,et al.  A tutorial on conformal prediction , 2007, J. Mach. Learn. Res..

[4]  Mark J. F. Gales,et al.  Incorporating Uncertainty into Deep Learning for Spoken Language Assessment , 2017, ACL.

[5]  Kit Po Wong,et al.  Optimal Prediction Intervals of Wind Power Generation , 2014, IEEE Transactions on Power Systems.

[6]  Graham Currie,et al.  Prediction intervals to account for uncertainties in neural network predictions: Methodology and application in bus travel time prediction , 2011, Eng. Appl. Artif. Intell..

[7]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[8]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[9]  Lian Yan,et al.  Predicting prostate cancer recurrence via maximizing the concordance index , 2004, KDD.

[10]  Abbas Khosravi,et al.  Uncertainty handling using neural network-based prediction intervals for electrical load forecasting , 2014 .

[11]  Xueying Sun,et al.  Prediction Interval Construction for Byproduct Gas Flow Forecasting Using Optimized Twin Extreme Learning Machine , 2017 .

[12]  Tom Heskes,et al.  Practical Confidence and Prediction Intervals , 1996, NIPS.

[13]  Enrico Zio,et al.  NSGA-II-trained neural network approach to the estimation of prediction intervals of scale deposition rate in oil & gas equipment , 2013, Expert Syst. Appl..

[14]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[15]  Michael Cogswell,et al.  Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks , 2015, ArXiv.

[16]  Inés María Galván,et al.  Multi-objective evolutionary optimization of prediction intervals for solar energy forecasting with neural networks , 2017, Inf. Sci..

[17]  Jidong Wang,et al.  Wind Power Interval Prediction Based on Improved PSO and BP Neural Network , 2017 .

[18]  Amir F. Atiya,et al.  Lower Upper Bound Estimation Method for Construction of Neural Network-Based Prediction Intervals , 2011, IEEE Transactions on Neural Networks.

[19]  Ryan P. Adams,et al.  Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks , 2015, ICML.

[20]  Amir F. Atiya,et al.  Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances , 2011, IEEE Transactions on Neural Networks.

[21]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[22]  Alan F. Murray,et al.  Confidence estimation methods for neural networks : a practical comparison , 2001, ESANN.

[23]  Karl O. Jones.  COMPARISON OF GENETIC ALGORITHM AND PARTICLE SWARM OPTIMISATION , 2005 .

[24]  Zhigang Zeng,et al.  Landslide Displacement Prediction With Uncertainty Based on Neural Networks With Random Hidden Weights , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Robert Tibshirani,et al.  A Comparison of Some Error Estimates for Neural Network Models , 1996, Neural Computation.

[26]  A. Weigend,et al.  Estimating the mean and variance of the target probability distribution , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[27]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[28]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[29]  Jie Chen,et al.  Wind Power Forecasting Using Multi-Objective Evolutionary Algorithms for Wavelet Neural Network-Optimized Prediction Intervals , 2018 .