Neural Network-Based Uncertainty Quantification: A Survey of Methodologies and Applications

Uncertainty quantification plays a critical role in the process of decision making and optimization in many fields of science and engineering. The field has gained an overwhelming attention among researchers in recent years resulting in an arsenal of different methods. Probabilistic forecasting and in particular prediction intervals (PIs) are one of the techniques most widely used in the literature for uncertainty quantification. Researchers have reported studies of uncertainty quantification in critical applications such as medical diagnostics, bioinformatics, renewable energies, and power grids. The purpose of this survey paper is to comprehensively study neural network-based methods for construction of prediction intervals. It will cover how PIs are constructed, optimized, and applied for decision-making in presence of uncertainties. Also, different criteria for unbiased PI evaluation are investigated. The paper also provides some guidelines for further research in the field of neural network-based uncertainty quantification.

[1]  Mansun Chan,et al.  Modeling of fringe current for semiconductor-extended organic TFTs , 2016, 2016 IEEE International Conference on Electron Devices and Solid-State Circuits (EDSSC).

[2]  Amir F. Atiya,et al.  Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances , 2011, IEEE Transactions on Neural Networks.

[3]  Vania Almeida,et al.  Prediction intervals for electric load forecast: Evaluation for different profiles , 2015, 2015 18th International Conference on Intelligent System Application to Power Systems (ISAP).

[4]  Mukta Paliwal,et al.  Neural networks and statistical techniques: A review of applications , 2009, Expert Syst. Appl..

[5]  Nadarajah Mithulananthan,et al.  PV based EV charging at universities using supplied historical PV output ramp , 2018 .

[6]  Liping Xie,et al.  Direct interval forecasting of wind speed using radial basis function neural networks in a multi-objective optimization framework , 2016, Neurocomputing.

[7]  Irena Koprinska,et al.  Forecasting hourly electricity load profile using neural networks , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[8]  Zongxiang Lu,et al.  Probabilistic short-term wind power forecasting based on deep neural networks , 2016, 2016 International Conference on Probabilistic Methods Applied to Power Systems (PMAPS).

[9]  Saeid Nahavandi,et al.  A prediction interval-based approach to determine optimal structures of neural network metamodels , 2010, Expert Syst. Appl..

[10]  Kwok-Wing Chau,et al.  ANN-based interval forecasting of streamflow discharges using the LUBE method and MOFIPS , 2015, Eng. Appl. Artif. Intell..

[11]  John Bjørnar Bremnes,et al.  Probabilistic wind power forecasts using local quantile regression , 2004 .

[12]  Carlos F.M. Coimbra,et al.  Real-time prediction intervals for intra-hour DNI forecasts , 2015 .

[13]  Inés María Galván,et al.  Multi-objective evolutionary optimization of prediction intervals for solar energy forecasting with neural networks , 2017, Inf. Sci..

[14]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[15]  Paras Mandal,et al.  A Hybrid Intelligent Model for Deterministic and Quantile Regression Approach for Probabilistic Wind Power Forecasting , 2014, IEEE Transactions on Power Systems.

[16]  Tao Lu,et al.  Prediction of indoor temperature and relative humidity using neural network models: model comparison , 2009, Neural Computing and Applications.

[17]  W. Rivera,et al.  Short term wind speed forecasting in La Venta, Oaxaca, México, using artificial neural networks , 2009 .

[18]  Saeid Nahavandi,et al.  Closure to the Discussion of “Prediction Intervals for Short-Term Wind Farm Generation Forecasts” and “Combined Nonparametric Prediction Intervals for Wind Power Generation” and the Discussion of “Combined Nonparametric Prediction Intervals for Wind Power Generation” , 2014 .

[19]  Saeid Nahavandi,et al.  An optimized mean variance estimation method for uncertainty quantification of wind power forecasts , 2014 .

[20]  Michael Negnevitsky,et al.  Wind speed forecast model for wind farm based on a hybrid machine learning algorithm , 2015 .

[21]  Pedro Vit,et al.  Interval prediction for a Poisson process , 1973 .

[22]  Aidong Adam Ding,et al.  Backpropagation of pseudo-errors: neural networks that are adaptive to heterogeneous noise , 2003, IEEE Trans. Neural Networks.

[23]  Felipe Valencia,et al.  Prediction interval based on type-2 fuzzy systems for wind power generation and loads in microgrid control design , 2016, 2016 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE).

[24]  Saeid Nahavandi,et al.  Construction of Optimal Prediction Intervals for Load Forecasting Problems , 2010, IEEE Transactions on Power Systems.

[25]  Zhenhuan Gong,et al.  PRESS: PRedictive Elastic ReSource Scaling for cloud systems , 2010, 2010 International Conference on Network and Service Management.

[26]  Peter Auer,et al.  Using Confidence Bounds for Exploitation-Exploration Trade-offs , 2003, J. Mach. Learn. Res..

[27]  Hana Ahmad Sartawi,et al.  Bayesian prediction bounds for the burr type X model , 1991 .

[28]  A. Weigend,et al.  Estimating the mean and variance of the target probability distribution , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[29]  Kit Po Wong,et al.  Optimal Prediction Intervals of Wind Power Generation , 2014, IEEE Transactions on Power Systems.

[30]  R. Stine Bootstrap Prediction Intervals for Regression , 1985 .

[31]  Yonghua Song,et al.  Nonparametric Prediction Intervals of Wind Power via Linear Programming , 2018, IEEE Transactions on Power Systems.

[32]  Francesco Giordano,et al.  Forecasting nonlinear time series with neural network sieve bootstrap , 2007, Comput. Stat. Data Anal..

[33]  Kashem M. Muttaqi,et al.  Sustainable energy system design with distributed renewable resources considering economic, environmental and uncertainty aspects , 2015 .

[34]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[35]  Saeid Nahavandi,et al.  Prediction interval-based neural network modelling of polystyrene polymerization reactor – A new perspective of data-based modelling , 2014 .

[36]  Padraig Cunningham,et al.  Confidence and prediction intervals for neural network ensembles , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[37]  John P. Parmigiani,et al.  An experimental assessment of methods to predict crack deflection at an interface , 2017 .

[38]  Xun Gong,et al.  Bootstrap prediction interval estimation for wind speed forecasting , 2015, 2015 IEEE Energy Conversion Congress and Exposition (ECCE).

[39]  J. Richard Landis Introductory statistical analysis , 1974 .

[40]  Abbas Khosravi,et al.  Short-Term Load and Wind Power Forecasting Using Neural Network-Based Prediction Intervals , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[41]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[42]  Saeid Nahavandi,et al.  Constructing Optimal Prediction Intervals by Using Neural Networks and Bootstrap Method , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[43]  Carlos E. Pedreira,et al.  Neural networks for short-term load forecasting: a review and evaluation , 2001 .

[44]  Saeid Nahavandi,et al.  Quantifying uncertainties of neural network-based electricity price forecasts , 2013 .

[45]  Tomonobu Senjyu,et al.  A new strategy to quantify uncertainties of wavelet-GRNN-PSO based solar PV power forecasts using bootstrap confidence intervals , 2015, 2015 IEEE Power & Energy Society General Meeting.

[46]  Raymond K. Wong,et al.  Currency Exchange Rate Forecasting From News Headlines , 2002, Australasian Database Conference.

[47]  Saeid Nahavandi,et al.  Improving the Quality of Prediction Intervals Through Optimal Aggregation , 2015, IEEE Transactions on Industrial Electronics.

[48]  Ronald J. Wonnacott,et al.  Trade and Investment in a Hub‐and‐Spoke System Versus a Free Trade Area , 1996 .

[49]  H. J. Van Zuylen,et al.  Bayesian committee of neural networks to predict travel times with confidence intervals , 2009 .

[50]  Jakub Nowotarski,et al.  LTSCWAVE: MATLAB function to estimate and forecast the long-term seasonal component (LTSC) of an electricity spot price series using wavelet-based methods , 2013 .

[51]  Robert Tibshirani,et al.  A Comparison of Some Error Estimates for Neural Network Models , 1996, Neural Computation.

[52]  J. Robinson,et al.  Sequential choice of an optimal dose: A prediction intervals approach , 1978 .

[53]  M. Elena Martinez-Perez,et al.  Retinal degeneration in children: Dark adapted visual threshold and arteriolar diameter , 2008, Vision Research.

[54]  Mohammad Anwar Hosen Prediction interval-based modelling and control of nonlinear processes , 2015 .

[55]  Luren Yang,et al.  An Evaluation of Confidence Bound Estimation Methods for Neural Networks , 2002, Advances in Computational Intelligence and Learning.

[56]  Keith Worden,et al.  Uncertainty analysis of a neural network used for fatigue lifetime prediction , 2008 .

[57]  Simon J. Mason,et al.  Comparison of Some Statistical Methods of Probabilistic Forecasting of ENSO. , 2002 .

[58]  Nadarajah Mithulananthan,et al.  A Day-Ahead Forecasting Model for Probabilistic EV Charging Loads at Business Premises , 2018, IEEE Transactions on Sustainable Energy.

[59]  Yanqing Zhang,et al.  Statistical fuzzy interval neural networks for currency exchange rate time series prediction , 2007, Appl. Soft Comput..

[60]  Chris Chatfield,et al.  Prediction Intervals for Time-Series Forecasting , 2001 .

[61]  Hussain Mohammed Dipu Kabir,et al.  A Resilient Auction Framework for Deadline-Aware Jobs in Cloud Spot Market , 2017, 2017 IEEE 36th Symposium on Reliable Distributed Systems (SRDS).

[62]  Paras Mandal,et al.  Solar PV power generation forecast using a hybrid intelligent approach , 2013, 2013 IEEE Power & Energy Society General Meeting.

[63]  Chris Chatfield,et al.  Calculating Interval Forecasts , 1993 .

[64]  Farouq S Mjalli,et al.  Use of artificial neural network black-box modeling for the prediction of wastewater treatment plants performance. , 2007, Journal of environmental management.

[65]  Jianxue Wang,et al.  Review on probabilistic forecasting of wind power generation , 2014 .

[66]  K. P. Sudheer,et al.  Comparison of methods used for quantifying prediction interval in artificial neural network hydrologic models , 2016, Modeling Earth Systems and Environment.

[67]  Bijaya K. Panigrahi,et al.  Prediction interval estimation for electricity price and demand using support vector machines , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[68]  Ashok N. Srivistava,et al.  Bootstrap Prediction Intervals in Non-Parametric Regression with Applications to Anomaly Detection , 2012 .

[69]  Pierre Pinson,et al.  Discussion of “Prediction Intervals for Short-Term Wind Farm Generation Forecasts” and “Combined Nonparametric Prediction Intervals for Wind Power Generation” , 2014 .

[70]  Michael P. Clements,et al.  Evaluating interval forecasts of high-frequency financial data , 2003 .

[71]  M. Valipour Optimization of neural networks for precipitation analysis in a humid region to detect drought and wet year alarms , 2016 .

[72]  R. D. Veaux,et al.  Prediction intervals for neural networks via nonlinear regression , 1998 .

[73]  Z. Dong,et al.  A Statistical Approach for Interval Forecasting of the Electricity Price , 2008, IEEE Transactions on Power Systems.

[74]  F. Fisher Tests of Equality Between Sets of Coefficients in Two Linear Regressions: An Expository Note , 1970 .

[75]  Peter F. Christoffersen Evaluating Interval Forecasts , 1998 .

[76]  David Hinkley,et al.  Bootstrap Methods: Another Look at the Jackknife , 2008 .

[77]  J. Felsenstein CONFIDENCE LIMITS ON PHYLOGENIES: AN APPROACH USING THE BOOTSTRAP , 1985, Evolution; international journal of organic evolution.

[78]  Bijaya K. Panigrahi,et al.  Prediction Interval Estimation of Electricity Prices Using PSO-Tuned Support Vector Machines , 2015, IEEE Transactions on Industrial Informatics.

[79]  J. T. Hwang,et al.  Prediction Intervals for Artificial Neural Networks , 1997 .

[80]  Irena Koprinska,et al.  Forecasting solar power generated by grid connected PV systems using ensembles of neural networks , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[81]  R. Weron A look into the future of electricity (spot) price forecasting , 2014 .

[82]  Saeid Nahavandi,et al.  Partial Adversarial Training for Prediction Interval , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[83]  Tom Schaul,et al.  Unifying Count-Based Exploration and Intrinsic Motivation , 2016, NIPS.

[84]  Zhao Xu,et al.  Direct Interval Forecasting of Wind Power , 2013, IEEE Transactions on Power Systems.

[85]  E. Zio,et al.  A study of the bootstrap method for estimating the accuracy of artificial neural networks in predicting nuclear transient processes , 2006, IEEE Transactions on Nuclear Science.

[86]  Adnan M. Awad,et al.  Large sample prediction intervals for a future sample mean: A comparative study , 1986 .

[87]  Jakub Nowotarski,et al.  Computing electricity spot price prediction intervals using quantile regression and forecast averaging , 2015, Comput. Stat..

[88]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[89]  Hiroshi Nishiura Early Detection of Nosocomial Outbreaks Caused by Rare Pathogens: A Case Study Employing Score Prediction Interval , 2012, Osong public health and research perspectives.

[90]  S. Roberts,et al.  Confidence Intervals and Prediction Intervals for Feed-Forward Neural Networks , 2001 .

[91]  Hyun Myung,et al.  Cooperative Particle Swarm Optimization-Based Predictive Controller for Multi-robot Formation , 2012, IAS.

[92]  Gerald J. Hahn,et al.  Factors for Calculating Two-Sided Prediction Intervals for Samples from a Normal Distribution , 1969 .

[93]  Zainal Ahmad,et al.  Modelling and control of different types of polymerization processes using neural networks technique: A review , 2010 .

[94]  Abbas Khosravi Construction and optimisation of prediction intervals for neural network models , 2010 .

[95]  Anuj Abraham,et al.  Very Short Term Wind Power Forecasting Using PSO-Neural Network Hybrid System , 2011, ACC.

[96]  Abbas Khosravi,et al.  Uncertainty handling using neural network-based prediction intervals for electrical load forecasting , 2014 .

[97]  George Stavrakakis,et al.  Wind power forecasting using advanced neural networks models , 1996 .

[98]  Yi-Chung Hu,et al.  Comparing four bankruptcy prediction models: Logit, quadratic interval logit, neural and fuzzy neural networks , 2010, Expert Syst. Appl..

[99]  Yonggang Wu,et al.  An Advanced Approach for Construction of Optimal Wind Power Prediction Intervals , 2015, IEEE Transactions on Power Systems.

[100]  Shen Liu,et al.  Beyond point forecasting: evaluation of alternative prediction intervals for tourist arrivals , 2011 .

[101]  Jin Lin,et al.  Direct Quantile Regression for Nonparametric Probabilistic Forecasting of Wind Power Generation , 2017, IEEE Transactions on Power Systems.

[102]  Ruddy Blonbou,et al.  Very short-term wind power forecasting with neural networks and adaptive Bayesian learning , 2011 .

[103]  Michael P. Clements,et al.  Evaluating the Bank of England Density Forecasts of Inflation , 2004 .

[104]  Zhang Yan,et al.  A review on the forecasting of wind speed and generated power , 2009 .

[105]  Saeid Nahavandi,et al.  Prediction interval with examples of similar pattern and prediction strength , 2017, 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE).

[106]  Saeid Nahavandi,et al.  Improving Prediction Interval Quality: A Genetic Algorithm-Based Method Applied to Neural Networks , 2009, ICONIP.

[107]  Samuel D. Stearns Arithmetic coding in lossless waveform compression , 1995, IEEE Trans. Signal Process..

[108]  Kit Po Wong,et al.  Probabilistic Forecasting of Wind Power Generation Using Extreme Learning Machine , 2014, IEEE Transactions on Power Systems.

[109]  Kit Po Wong,et al.  Discussion of “Combined Nonparametric Prediction Intervals for Wind Power Generation” , 2014 .

[110]  Amir F. Atiya,et al.  Lower Upper Bound Estimation Method for Construction of Neural Network-Based Prediction Intervals , 2011, IEEE Transactions on Neural Networks.

[111]  R. Weron Modeling and Forecasting Electricity Loads and Prices: A Statistical Approach , 2006 .

[112]  Torsten Hothorn,et al.  Prediction intervals for future BMI values of individual children - a non-parametric approach by quantile boosting , 2012, BMC Medical Research Methodology.

[113]  Pierre Pinson,et al.  Very-Short-Term Probabilistic Wind Power Forecasts by Sparse Vector Autoregression , 2016, IEEE Transactions on Smart Grid.

[114]  D. Heien,et al.  The Demand for Dairy Products: Structure, Prediction, and Decomposition , 1988 .

[115]  Israel Saguy,et al.  STATISTICAL EVALUATION OF ARRHENIUS MODEL AND ITS APPLICABILITY IN PREDICTION OF FOOD QUALITY LOSSES , 1985 .

[116]  O. B. Chedzoy,et al.  Introductory Statistical Analysis , 1975 .

[117]  Thong Ngee Goh,et al.  Neural network modeling with confidence bounds: a case study on the solder paste deposition process , 2001 .

[118]  Mohd Ali Hashim,et al.  Prediction of deep eutectic solvents densities at different temperatures , 2011 .

[119]  Kostas S. Metaxiotis,et al.  Artificial intelligence in short term electric load forecasting: a state-of-the-art survey for the researcher , 2003 .

[120]  Javier Arroyo Gallardo,et al.  Different approaches to forecast interval time series: a comparison in finance , 2011 .

[121]  M Jones,et al.  Accuracy of point predictions in survival analysis , 2001, Statistics in medicine.

[122]  Jagdish K. Patel,et al.  Prediction intervals - a review , 1989 .

[123]  Durga L. Shrestha,et al.  Machine learning approaches for estimation of prediction interval for the model output , 2006, Neural Networks.

[124]  Vincent Mok,et al.  Stroke risk assessment for the community by automatic retinal image analysis using fundus photograph , 2016 .

[125]  Tom Heskes,et al.  Practical Confidence and Prediction Intervals , 1996, NIPS.

[126]  Ying Wang,et al.  Optimal Wind Power Uncertainty Intervals for Electricity Market Operation , 2018, IEEE Transactions on Sustainable Energy.

[127]  P. Pinson,et al.  Wind power in electricity markets and the value of forecasting , 2017 .

[128]  John P. Parmigiani,et al.  Effect of incident angle on crack propagation at interfaces , 2016 .