A new optimization algorithm for non-stationary time series prediction based on recurrent neural networks

Abstract Deep neural network with recurrent structure was proposed in the recent years and has been applied to time series forecasting. Many optimization algorithms are developed under the assumption of invariant and stationary data distributions, which is invalid for the non-stationary data. A novel optimization algorithm for modeling non-stationary time series is proposed in this paper. A moving window and exponential decay weights are used in this algorithm to eliminate the effects of the history gradients. The regret bound of the new algorithm is analyzed to ensure the convergency of the calculation. Simulations are done on short-term power load data sets, which are typically non-stationary. The results are superior to the existing optimization algorithms.

[1]  Robert Jenssen,et al.  Recurrent Neural Networks for Short-Term Load Forecasting , 2017, SpringerBriefs in Computer Science.

[2]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[3]  Alex Graves,et al.  Supervised Sequence Labelling , 2012 .

[4]  Sheldon X.-D. Tan,et al.  FPGA-Based Implementation of a Multilayer Perceptron Suitable for Chaotic Time Series Prediction , 2018, Technologies.

[5]  Léon Bottou,et al.  On-line learning and stochastic approximations , 1999 .

[6]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[7]  Matthew D. Zeiler.  ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.

[8]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[9]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[10]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[11]  Sercan Ömer Arik,et al.  Deep Voice 2: Multi-Speaker Neural Text-to-Speech , 2017, NIPS.

[12]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[13]  Neil Davey,et al.  Time Series Prediction and Neural Networks , 2001, J. Intell. Robotic Syst..

[14]  J. Contreras,et al.  ARIMA models to predict next-day electricity prices , 2002 .

[15]  Guiming Luo,et al.  Short term power load prediction with knowledge transfer , 2015, Inf. Syst..

[16]  A. Zeevi,et al.  Non-Stationary Stochastic Optimization , 2014 .

[17]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.