Improved simple deterministically constructed Cycle Reservoir Network with Sensitive Iterative Pruning Algorithm

Abstract Reservoir Computing (RC) is an effective approach to design and train recurrent neural networks, which is successfully and widely applied in real-valued time series modeling tasks. However, RC has been criticized for not being principled enough, namely the reservoir which is unlikely to be optimal because the reservoir connectivity and weight structure are created randomly. A new Simple Cycle Reservoir Network (SCRN) with deterministically constructed connectivity and weight structure can yield performance competitive with standard Echo State Network (ESN). In order to determine the proper size of the reservoir and improve generalization ability of SCRN, a Sensitive Iterated Pruning Algorithm (SIPA), in which a larger than necessary reservoir is employed firstly and then its size is reduced by pruning out the least sensitive internal units, is proposed to optimize the reservoir size and weights of SCRN. A system identification and two time-series benchmark tasks are applied to demonstrate the feasibility and superiority of SIPA. The results show that the SIPA method significantly outperforms a Least Angle Regression (LAR) method and SIPA is able to improve the generalization performance of SCRN. Besides, two well known reservoir characterizations, i.e. pseudo-Lyapunov exponent of the reservoir dynamics and Memory Capacity, and the impact of SIPA on two characterizations are investigated.

[1]  Amir F. Atiya,et al.  New results on recurrent network training: unifying the algorithms and accelerating convergence , 2000, IEEE Trans. Neural Networks Learn. Syst..

[2]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[3]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[4]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[5]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[6]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[7]  Peter Tiño,et al.  Comparison of Echo State Networks with Simple Recurrent Networks and Variable-Length Markov Models on Symbolic Sequences , 2007, ICANN.

[8]  John G. Harris,et al.  Minimum mean squared error time series classification using an echo state network prediction model , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[9]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[10]  J.J. Steil,et al.  Backpropagation-decorrelation: online recurrent learning with O(N) complexity , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[11]  Garrison W. Cottrell,et al.  2007 Special Issue: Learning grammatical structure with Echo State Networks , 2007 .

[12]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[13]  D. Prokhorov,et al.  Echo state networks: appeal and challenges , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[14]  Sun-Yuan Kung,et al.  A delay damage model selection algorithm for NARX neural networks , 1997, IEEE Trans. Signal Process..

[15]  Herbert Jaeger,et al.  Echo state network , 2007, Scholarpedia.

[16]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[17]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[18]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[19]  Christopher MacLeod,et al.  Incremental growth in modular neural networks , 2009, Eng. Appl. Artif. Intell..

[20]  Shih-Hung Yang,et al.  An evolutionary constructive and pruning algorithm for artificial neural networks and its prediction applications , 2012, Neurocomputing.

[21]  C. L. Giles,et al.  Pruning recurrent neural networks for improved generalization performance , 1994, Proceedings of IEEE Workshop on Neural Networks for Signal Processing.

[22]  Hendrik Van Brussel,et al.  A first attempt of reservoir pruning for classification problems , 2007, ESANN.

[23]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[24]  Caihong Li,et al.  Multi-steps prediction of chaotic time series based on echo state network , 2010, 2010 IEEE Fifth International Conference on Bio-Inspired Computing: Theories and Applications (BIC-TA).

[25]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[26]  Antonio Luchetta.  Automatic generation of the optimum threshold for parameter weighted pruning in multiple heterogeneous output neural networks , 2008, Neurocomputing.

[27]  Tao Chen,et al.  Condition prediction of flue gas turbine based on Echo State Network , 2010, 2010 Sixth International Conference on Natural Computation.

[28]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[29]  Jürgen Schmidhuber,et al.  Training Recurrent Networks by Evolino , 2007, Neural Computation.

[30]  Giovanna Castellano,et al.  An iterative pruning algorithm for feedforward neural networks , 1997, IEEE Trans. Neural Networks.

[31]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[32]  Ralph Kennel,et al.  Levenberg-marquardt-based OBS algorithm using adaptive pruning interval for system identification with dynamic neural networks , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[33]  Peng Yu,et al.  Clustered complex echo state networks for traffic forecasting with prior knowledge , 2011, 2011 IEEE International Instrumentation and Measurement Technology Conference.

[34]  Jochen J. Steil,et al.  Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning , 2007, Neural Networks.

[35]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[36]  Zehong Yang,et al.  Short-term stock price prediction based on echo state networks , 2009, Expert Syst. Appl..

[37]  J. Sum,et al.  Extended Kalman FilterBased Pruning Method for Recurrent Neural Networks , 1998, Neural Computation.

[38]  Mahdi Vasighi,et al.  Genetic Algorithms for architecture optimisation of Counter-Propagation Artificial Neural Networks , 2011 .