Generating probabilistic predictions using mean-variance estimation and echo state network

In conventional time series prediction techniques, uncertainty associated with predictions are usually ignored. Probabilistic predictors, on the other hand, can measure the uncertainty in predictions, to provide better supports for decision-making processes. A dynamic probabilistic predictor, named as echo state mean-variance estimation (ESMVE) model, is proposed. The model is constructed with two recurrent neural networks. These networks are trained into a mean estimator and a variance estimator respectively, following the algorithm of echo state networks. ESMVE generate point predictions by estimating the means of a target time series, while it also measures the uncertainty in its predictions by generating variance estimations. Experiments conducted on synthetic data sets show advantages of ESMVE over MVE models constructed with static networks. Effectiveness of ESMVE in real world prediction tasks have also been verified in our case studies.

[1]  Amir F. Atiya,et al.  Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances , 2011, IEEE Transactions on Neural Networks.

[2]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[3]  Bellie Sivakumar,et al.  A phase-space reconstruction approach to prediction of suspended sediment concentration in rivers , 2002 .

[4]  J. B. Bremnes.  A comparison of a few statistical models for making quantile wind power forecasts , 2006 .

[5]  Benjamin Schrauwen,et al.  Analog readout for optical reservoir computers , 2012, NIPS.

[6]  Yiannis Demiris,et al.  Echo State Gaussian Process , 2011, IEEE Transactions on Neural Networks.

[7]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[8]  S. Fan,et al.  Short-term load forecasting based on an adaptive hybrid method , 2006, IEEE Transactions on Power Systems.

[9]  Min Han,et al.  Prediction of chaotic time series based on the recurrent predictor neural network , 2004, IEEE Transactions on Signal Processing.

[10]  M. Xia,et al.  Deformation and mechanism of landslide influenced by the effects of reservoir water and rainfall, Three Gorges, China , 2013, Natural Hazards.

[11]  Jie Zhang,et al.  Wind Power Ramp Event Forecasting Using a Stochastic Scenario Generation Method , 2015, IEEE Transactions on Sustainable Energy.

[12]  Zhigang Zeng,et al.  Training enhanced reservoir computing predictor for landslide displacement , 2015 .

[13]  Pericles A. Mitkas,et al.  Adaptive reservoir computing through evolution and learning , 2013, Neurocomputing.

[14]  Suzanne Lacasse,et al.  Displacement prediction in colluvial landslides, Three Gorges Reservoir, China , 2013, Landslides.

[15]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[16]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[17]  Jürgen Schmidhuber,et al.  Training Recurrent Networks by Evolino , 2007, Neural Computation.

[18]  Amir F. Atiya,et al.  Lower Upper Bound Estimation Method for Construction of Neural Network-Based Prediction Intervals , 2011, IEEE Transactions on Neural Networks.

[19]  Benjamin Schrauwen,et al.  Recurrent Kernel Machines: Computing with Infinite Echo State Networks , 2012, Neural Computation.

[20]  Russell Y. Webb,et al.  Reservoir Computing for Prediction of the Spatially-Variant Point Spread Function , 2008, IEEE Journal of Selected Topics in Signal Processing.

[21]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[22]  S. Roberts,et al.  Confidence Intervals and Prediction Intervals for Feed-Forward Neural Networks , 2001 .

[23]  Amy Loutfi,et al.  A review of unsupervised feature learning and deep learning for time-series modeling , 2014, Pattern Recognit. Lett..

[24]  Yiannis Demiris,et al.  The copula echo state network , 2012, Pattern Recognit..

[25]  George E. P. Box,et al.  Time Series Analysis: Forecasting and Control , 1977 .

[26]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[27]  Benjamin Schrauwen,et al.  An overview of reservoir computing: theory, applications and implementations , 2007, ESANN.

[28]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[29]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[30]  Amaury Lendasse,et al.  Autoregressive time series prediction by means of fuzzy inference systems using nonparametric residual variance estimation , 2010, Fuzzy Sets Syst..

[31]  Hava T. Siegelmann,et al.  The Dynamic Universality of Sigmoidal Neural Networks , 1996, Inf. Comput..

[32]  Yuichi Nakamura,et al.  Approximation of dynamical systems by continuous time recurrent neural networks , 1993, Neural Networks.

[33]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Peng Hong,et al.  Multi-step-prediction of chaotic time series based on co-evolutionary recurrent neural network , 2008 .

[35]  Okyay Kaynak,et al.  Grey system theory-based models in time series prediction , 2010, Expert Syst. Appl..

[36]  A. Weigend,et al.  Estimating the mean and variance of the target probability distribution , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[37]  Min Han,et al.  Global mutual information-based feature selection approach using single-objective and multi-objective optimization , 2015, Neurocomputing.

[38]  Z. Zeng,et al.  Extreme learning machine for the displacement prediction of landslide under rainfall and reservoir level , 2014, Stochastic Environmental Research and Risk Assessment.

[39]  Tingwen Huang,et al.  Distributed parameter estimation in unreliable sensor networks via broadcast gossip algorithms , 2016, Neural Networks.

[40]  Amir F. Atiya,et al.  Forecast combinations of computational intelligence and linear models for the NN5 time series forecasting competition , 2011 .

[41]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[42]  Kit Po Wong,et al.  A Hybrid Approach for Probabilistic Forecasting of Electricity Price , 2014, IEEE Transactions on Smart Grid.