Reinforced Two-Step-Ahead Weight Adjustment Technique for Online Training of Recurrent Neural Networks

A reliable forecast of future events possesses great value. The main purpose of this paper is to propose an innovative learning technique for reinforcing the accuracy of two-step-ahead (2SA) forecasts. The real-time recurrent learning (RTRL) algorithm for recurrent neural networks (RNNs) can effectively model the dynamics of complex processes and has been used successfully in one-step-ahead forecasts for various time series. A reinforced RTRL algorithm for 2SA forecasts using RNNs is proposed in this paper, and its performance is investigated by two famous benchmark time series and a streamflow during flood events in Taiwan. Results demonstrate that the proposed reinforced 2SA RTRL algorithm for RNNs can adequately forecast the benchmark (theoretical) time series, significantly improve the accuracy of flood forecasts, and effectively reduce time-lag effects.

[1]  Xiangping Zeng,et al.  Low-Complexity Nonlinear Adaptive Filter Based on a Pipelined Bilinear Recurrent Neural Network , 2011, IEEE Transactions on Neural Networks.

[2]  Yen-Ming Chiang,et al.  Dynamic neural networks for real-time water level predictions of sewerage systems-covering gauged and ungauged sites , 2010 .

[3]  Jun Wang,et al.  Recurrent neural networks for nonlinear output regulation , 2001, Autom..

[4]  Li-Chiu Chang,et al.  Auto-configuring radial basis function networks for chaotic time series and flood forecasting , 2009 .

[5]  Bellie Sivakumar,et al.  River flow forecasting: use of phase-space reconstruction and artificial neural networks approaches , 2002 .

[6]  S. Thomas Ng,et al.  A Modified Neural Network for Improving River Flow Prediction/Un Réseau de Neurones Modifié pour Améliorer la Prévision de L'Écoulement Fluvial , 2005 .

[7]  Bhaskar D. Rao,et al.  On-line learning algorithms for locally recurrent neural networks , 1999, IEEE Trans. Neural Networks.

[8]  José Carlos Príncipe,et al.  Water Inflow Forecasting using the Echo State Network: a Brazilian Case Study , 2007, 2007 International Joint Conference on Neural Networks.

[9]  Jing Peng,et al.  An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories , 1990, Neural Computation.

[10]  Liang Li,et al.  Nonlinear adaptive prediction of nonstationary signals , 1995, IEEE Trans. Signal Process..

[11]  Lida Xu,et al.  A new type of recurrent fuzzy neural network for modeling dynamic systems , 2001, Knowl. Based Syst..

[12]  Gürsel Serpen,et al.  Simultaneous recurrent neural network trained with non-recurrent backpropagation algorithm for static optimisation , 2003, Neural Computing & Applications.

[13]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[14]  Hongjian Zhang,et al.  Time-delay neural network for the prediction of carbonation tower's temperature , 2003, IEEE Trans. Instrum. Meas..

[15]  Mohamed Chtourou,et al.  A hybrid approach for training recurrent neural networks: application to multi-step-ahead prediction of noisy and large data sets , 2007, Neural Computing and Applications.

[16]  L. Cao.  Practical method for determining the minimum embedding dimension of a scalar time series , 1997 .

[17]  Kotaro Hirasawa,et al.  Universal learning network and its application to robust control , 2000, IEEE Trans. Syst. Man Cybern. Part B.

[18]  Fraser,et al.  Independent coordinates for strange attractors from mutual information. , 1986, Physical review. A, General physics.

[19]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[20]  L-C. Chang,et al.  Real-Time Reservoir Operation for Flood Control Using Artificial Intelligent Techniques , 2010 .

[21]  B. Bobée,et al.  Multivariate Reservoir Inflow Forecasting Using Temporal Neural Networks , 2001 .

[22]  Qing Song.  Robust Initialization of a Jordan Network With Recurrent Constrained Learning , 2011, IEEE Transactions on Neural Networks.

[23]  T. Kerh,et al.  Treatment and Assessment of Nonlinear Seismic Data by a Genetic Algorithm Based Neural Network Model , 2009 .

[24]  Hong Peng,et al.  Identification of Dynamic Systems Using Recurrent Fuzzy Wavelet Network , 2006, ISNN.

[25]  Yen-Ming Chiang,et al.  A two‐step‐ahead recurrent neural network for stream‐flow forecasting , 2004 .

[26]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[27]  Li-Chiu Chang,et al.  Real‐time recurrent learning neural network for stream‐flow forecasting , 2002 .

[28]  Jie Zhang,et al.  Improving long range prediction for nonlinear process modelling through combining multiple neural networks , 2002, Proceedings of the International Conference on Control Applications.

[29]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[30]  Fi-John Chang,et al.  Evolutionary artificial neural networks for hydrological systems forecasting , 2009 .

[31]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[32]  Amir F. Atiya,et al.  Multi-step-ahead prediction using dynamic recurrent neural networks , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[33]  Sameer Sharma,et al.  Neural Network Models for Air Quality Prediction: A Comparative Study , 2007 .

[34]  Lifeng Xi,et al.  Transient Air-Fuel Ratio Estimation in Spark Ignition Engine Using Recurrent Neural Networks , 2007, KES.

[35]  Juebang Yu,et al.  Using recurrent neural network for adaptive predistortion linearization of RF amplifiers , 2002 .

[36]  Hubert Cardot,et al.  Study of the Behavior of a New Boosting Algorithm for Recurrent Neural Networks , 2005, ICANN.

[37]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[38]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[39]  David R. Maidment,et al.  Handbook of Hydrology , 1993 .

[40]  Khashayar Khorasani,et al.  Adaptive time delay neural network structures for nonlinear system identification , 2002, Neurocomputing.

[41]  Johan A. K. Suykens,et al.  Improved Long-Term Temperature Prediction by Chaining of Neural Networks , 2001, Int. J. Neural Syst..

[42]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[43]  Babak Nadjar Araabi,et al.  Predicting Chaotic Time Series Using Neural and Neurofuzzy Models: A Comparative Study , 2006, Neural Processing Letters.

[44]  Manuel Mazo,et al.  Using a New Model of Recurrent Neural Network for Control , 2004, Neural Processing Letters.

[45]  Jiaqiu Wang,et al.  Application of a Dynamic Recurrent Neural Network in Spatio-Temporal Forecasting , 2007, IF&GIS.

[46]  Kwok-wing Chau,et al.  A hybrid adaptive time-delay neural network model for multi-step-ahead prediction of sunspot activity , 2006 .

[47]  Kuolin Hsu,et al.  Estimation of physical variables from multichannel remotely sensed imagery using a neural network: Application to rainfall estimation , 1999 .

[48]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[49]  Caihong Li,et al.  Multi-steps prediction of chaotic time series based on echo state network , 2010, 2010 IEEE Fifth International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA).

[50]  J. Nash,et al.  River flow forecasting through conceptual models part I — A discussion of principles☆ , 1970 .

[51]  Saeed Zolfaghari,et al.  Chaotic time series prediction with residual analysis method using hybrid Elman-NARX neural networks , 2010, Neurocomputing.

[52]  Tiesong Hu,et al.  A Modified Neural Network for Improving River Flow Prediction , 2005 .

[53]  Jean Pierre Asselin de Beauville,et al.  Learning long-term dependencies by the selective addition of time-delayed connections to recurrent neural networks , 2002, Neurocomputing.

[54]  K. P. Unnikrishnan,et al.  Alopex: A Correlation-Based Learning Algorithm for Feedforward and Recurrent Neural Networks , 1994, Neural Computation.

[55]  Nikolay I. Nikolaev,et al.  Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling , 2010, IEEE Transactions on Neural Networks.

[56]  Michael R. Davenport,et al.  Continuous-time temporal back-propagation with adaptable time delays , 1993, IEEE Trans. Neural Networks.

[57]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[58]  Yen-Ming Chiang,et al.  Dynamic ANN for precipitation estimation and forecasting from radar observations , 2007 .

[59]  David Zipser,et al.  A Subgrouping Strategy that Reduces Complexity and Speeds Up Learning in Recurrent Networks , 1989, Neural Computation.

[60]  Bernd Freisleben,et al.  Neural Network Model Selection for Financial Time Series Prediction , 2001, Comput. Stat..