Recurrent neural network system using probability graph model optimization

We propose a novel recurrent neural network model based on a combination of the echo state network (ESN) and the dynamic Bayesian network (DBN). Our contribution includes the following: (1) A new graph-based echo state network (GESN) model is presented for nonlinear system modeling. The GESN consists of four layers: an input layer, reservoir layer, graph model optimization layer and output layer. Unlike the original ESN model, the GESN model uses probability graph inference to optimize reservoir states, and the optimized GESN is more robust. (2) A novel DBN graph is proposed for describing random probability signals in the GESN. Simultaneously, an inference algorithm of the DBN is deduced according to Bayesian rules and probability graph theory. Finally, GESN performance is tested using the Mackey–Glass time series and laser time series data forecasting The simulations and comparison results show that the proposed model is promising.

[1]  Çagdas Hakan Aladag,et al.  Fuzzy-time-series network used to forecast linear and nonlinear time series , 2015, Applied Intelligence.

[2]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[3]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[4]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[5]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[6]  Mantas Lukosevicius,et al.  Reservoir Computing and Self-Organized Neural Hierarchies , 2012 .

[7]  Jun Wang,et al.  Chaotic Time Series Prediction Based on a Novel Robust Echo State Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[9]  Roger Labahn,et al.  Design Strategies for Weight Matrices of Echo State Networks , 2012, Neural Computation.

[10]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[11]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[12]  Erol Egrioglu,et al.  A modified genetic algorithm for forecasting fuzzy time series , 2014, Applied Intelligence.

[13]  J. Gross,et al.  Graph Theory and Its Applications , 1998 .

[14]  Jonathan L. Gross,et al.  Graph Theory and Its Applications, Second Edition (Discrete Mathematics and Its Applications) , 2005 .

[15]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[16]  Hans-Jürgen Zimmermann,et al.  Fuzzy Set Theory - and Its Applications , 1985 .

[17]  Min Han,et al.  Prediction of chaotic time series based on the recurrent predictor neural network , 2004, IEEE Transactions on Signal Processing.

[18]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[19]  Habib Rostami,et al.  SCESN, SPESN, SWESN: Three recurrent neural echo state networks with clustered reservoirs for prediction of nonlinear and chaotic time series , 2015, Applied Intelligence.

[20]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[21]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[22]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[23]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[24]  Mantas Lukoševičius,et al.  On self-organizing reservoirs and their hierarchies , 2010 .