Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances

This paper evaluates the four leading techniques proposed in the literature for construction of prediction intervals (PIs) for neural network point forecasts. The delta, Bayesian, bootstrap, and mean-variance estimation (MVE) methods are reviewed and their performance for generating high-quality PIs is compared. PI-based measures are proposed and applied for the objective and quantitative assessment of each method's performance. A selection of 12 synthetic and real-world case studies is used to examine each method's performance for PI construction. The comparison is performed on the basis of the quality of generated PIs, the repeatability of the results, the computational requirements and the PIs variability with regard to the data uncertainty. The obtained results in this paper indicate that: 1) the delta and Bayesian methods are the best in terms of quality and repeatability, and 2) the MVE and bootstrap methods are the best in terms of low computational load and the width variability of PIs. This paper also introduces the concept of combinations of PIs, and proposes a new method for generating combined PIs using the traditional PIs. Genetic algorithm is applied for adjusting the combiner parameters through minimization of a PI-based cost function subject to two sets of restrictions. It is shown that the quality of PIs produced by the combiners is dramatically better than the quality of PIs obtained from each individual method.

[1]  Saeid Nahavandi,et al.  A genetic algorithm-based method for improving quality of travel time prediction intervals , 2011 .

[2]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[3]  S. Roberts,et al.  Confidence Intervals and Prediction Intervals for Feed-Forward Neural Networks , 2001 .

[4]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[5]  Keith Worden,et al.  Uncertainty analysis of a neural network used for fatigue lifetime prediction , 2008 .

[6]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[7]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[8]  R. D. Veaux,et al.  Prediction intervals for neural networks via nonlinear regression , 1998 .

[9]  Amir F. Atiya,et al.  Lower Upper Bound Estimation Method for Construction of Neural Network-Based Prediction Intervals , 2011, IEEE Transactions on Neural Networks.

[10]  Oleksandr Makeyev,et al.  Neural network with ensembles , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[11]  Z. Dong,et al.  A Statistical Approach for Interval Forecasting of the Electricity Price , 2008, IEEE Transactions on Power Systems.

[12]  H. J. Van Zuylen,et al.  Bayesian committee of neural networks to predict travel times with confidence intervals , 2009 .

[13]  Andrej Pázman,et al.  Nonlinear Regression , 2019, Handbook of Regression Analysis With Applications in R.

[14]  Robert Tibshirani,et al.  A Comparison of Some Error Estimates for Neural Network Models , 1996, Neural Computation.

[15]  A. Weigend,et al.  Estimating the mean and variance of the target probability distribution , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[16]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.

[17]  J. T. Hwang,et al.  Prediction Intervals for Artificial Neural Networks , 1997 .

[18]  Aidong Adam Ding,et al.  Backpropagation of pseudo-errors: neural networks that are adaptive to heterogeneous noise , 2003, IEEE Trans. Neural Networks.

[19]  Mohamed Azlan Hussain,et al.  Review of the applications of neural networks in chemical process control - simulation and online implementation , 1999, Artif. Intell. Eng..

[20]  Sevket I. Gunter.  Nonnegativity restricted least squares combinations , 1992 .

[21]  Tom Heskes,et al.  Practical Confidence and Prediction Intervals , 1996, NIPS.

[22]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[23]  Saeid Nahavandi,et al.  A prediction interval-based approach to determine optimal structures of neural network metamodels , 2010, Expert Syst. Appl..

[24]  Saeid Nahavandi,et al.  Prediction Intervals to Account for Uncertainties in Travel Time Prediction , 2011, IEEE Transactions on Intelligent Transportation Systems.

[25]  Rob J Hyndman,et al.  25 years of time series forecasting , 2006 .

[26]  Bimal K. Bose,et al.  Neural Network Applications in Power Electronics and Motor Drives—An Introduction and Perspective , 2007, IEEE Transactions on Industrial Electronics.

[27]  David J. C. MacKay,et al.  The Evidence Framework Applied to Classification Networks , 1992, Neural Computation.

[28]  Xin Yao,et al.  Evolving artificial neural network ensembles , 2008, IEEE Computational Intelligence Magazine.

[29]  James W. Taylor,et al.  Using combined forecasts with changing weights for electricity demand profiling , 2000, J. Oper. Res. Soc..

[30]  Knut Inge Fostervold,et al.  Maximum vs. minimum values: preferences of speakers and listeners for upper and lower limit estimates. , 2009, Acta psychologica.

[31]  Durga L. Shrestha,et al.  Machine learning approaches for estimation of prediction interval for the model output , 2006, Neural Networks.

[32]  Wenhua Liu,et al.  Forecasting the semiconductor industry cycles by bootstrap prediction intervals , 2007 .

[33]  Francesco Giordano,et al.  Forecasting nonlinear time series with neural network sieve bootstrap , 2007, Comput. Stat. Data Anal..

[34]  Saeid Nahavandi,et al.  Constructing prediction intervals for neural network metamodels of complex systems , 2009, 2009 International Joint Conference on Neural Networks.

[35]  Khashayar Khorasani,et al.  New training strategies for constructive neural networks with application to regression problems , 2004, Neural Networks.

[36]  Saeid Nahavandi,et al.  Construction of Optimal Prediction Intervals for Load Forecasting Problems , 2010, IEEE Transactions on Power Systems.

[37]  Debashis Kushary,et al.  Bootstrap Methods and Their Application , 2000, Technometrics.

[38]  Bart De Schutter,et al.  DAISY : A database for identification of systems , 1997 .

[39]  Colin R. Reeves,et al.  Genetic Algorithms: Principles and Perspectives: A Guide to Ga Theory , 2002 .

[40]  Dirk Van den Poel,et al.  Benefits of quantile regression for the analysis of customer lifetime value in a contractual setting: An application in financial services , 2009, Expert Syst. Appl..

[41]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[42]  Mukta Paliwal,et al.  Neural networks and statistical techniques: A review of applications , 2009, Expert Syst. Appl..

[43]  Goldberg,et al.  Genetic algorithms , 1993, Robust Control Systems with Genetic Algorithms.

[44]  Alan F. Murray,et al.  Confidence estimation methods for neural networks : a practical comparison , 2001, ESANN.

[45]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[46]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[47]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[48]  Peter Tino,et al.  IEEE Transactions on Neural Networks , 2009 .

[49]  Thong Ngee Goh,et al.  Neural network modeling with confidence bounds: a case study on the solder paste deposition process , 2001 .

[50]  Léon Personnaz,et al.  Construction of confidence intervals for neural networks based on least squares estimation , 2000, Neural Networks.

[51]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.