Design of polynomial echo state networks for time series prediction

Abstract Echo state networks (ESNs) have been widely used in the field of time series prediction. In conventional ESNs, the spectral radius of reservoir is always scaled to lower than 1 to satisfy the necessary condition for echo state property (ESP), while the sufficient condition is unregarded. Meanwhile, the output weights are always trained without considering the high order statistics of input signals. To solve above problems, the original ESN is extended to polynomial ESNs (PESNs) by employing the polynomial functions of input variables into output weights. Firstly, the reservoir of the PESN is built by the singular value decomposition (SVD) method. Secondly, the prime PESN (P-PESN) is implemented and its polynomial output weights are augmented by the high order statistics of inputs. Thirdly, the simplified PESN (S-PESN) is constructed by decomposing the polynomial output weights of the P-PESN into randomly generated polynomial nodes and tuned output weights. Furthermore, the regression matrix properties of P-PESN and S-PESN are theoretically analyzed, respectively. Finally, the simulation results show that the proposed PESNs obtain better performance than other methods in terms of prediction accuracy and learning speed.

[1]  John G. Harris,et al.  Automatic speech recognition using a predictive echo state network classifier , 2007, Neural Networks.

[2]  Ying Liu,et al.  Prediction for noisy nonlinear time series by echo state network based on dual estimation , 2012, Neurocomputing.

[3]  Ali Moeini,et al.  Evolutionary design of generalized polynomial neural networks for modelling and prediction of explosive forming process , 2005 .

[4]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[5]  Francesco Corona,et al.  Data-derived soft-sensors for biological wastewater treatment plants: An overview , 2013, Environ. Model. Softw..

[6]  Héctor Pomares,et al.  Soft-computing techniques and ARMA model for time series prediction , 2008, Neurocomputing.

[7]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[8]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[9]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[10]  Lin Lin,et al.  Genetic algorithm optimized double-reservoir echo state network for multi-regime time series prediction , 2017, Neurocomputing.

[11]  Min Han,et al.  Online multivariate time series prediction using SCKF-γESN model , 2015, Neurocomputing.

[12]  Meng Joo Er,et al.  Generalized Single-Hidden Layer Feedforward Networks for Regression Problems , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[13]  Xuefeng Yan,et al.  Optimizing the echo state network based on mutual information for modeling fed-batch bioprocesses , 2017, Neurocomputing.

[14]  Fuad E. Alsaadi,et al.  Event-triggered H ∞ state estimation for discrete-time stochastic genetic regulatory networks with Markovian jumping parameters and time-varying delays , 2016, Neurocomputing.

[15]  Andreas Zell,et al.  Optimizing recurrent reservoirs with neuro-evolution , 2016, Neurocomputing.

[16]  J. Laurin,et al.  Application of the Tikhonov regularization technique to the equivalent magnetic currents near-field technique , 2004, IEEE Transactions on Antennas and Propagation.

[17]  Junfei Qiao,et al.  An online self-adaptive modular neural network for time-varying systems , 2014, Neurocomputing.

[18]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[19]  Jun Wang,et al.  Chaotic Time Series Prediction Based on a Novel Robust Echo State Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[20]  Gene H. Golub,et al.  Matrix computations , 1983 .

[21]  Hong Qiao,et al.  Event-Triggered State Estimation for Discrete-Time Multidelayed Neural Networks With Stochastic Parameters and Incomplete Measurements , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Fernando José Von Zuben,et al.  Self-organization and lateral interaction in echo state network reservoirs , 2014, Neurocomputing.

[23]  P. Hansen.  The truncatedSVD as a method for regularization , 1987 .

[24]  Arash Miranian,et al.  Developing a Local Least-Squares Support Vector Machines-Based Neuro-Fuzzy Model for Nonlinear and Chaotic Time Series Prediction , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[26]  Zeng-Guang Hou,et al.  A hierarchical optimization neural network for large-scale dynamic systems , 2001, Autom..

[27]  Junfei Qiao,et al.  Growing Echo-State Network With Multiple Subreservoirs , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[28]  G. Venayagamoorthy,et al.  Reservoir based learning network for control of two-area power system with variable renewable generation , 2015, Neurocomputing.

[29]  Hossam Faris,et al.  Bidirectional reservoir networks trained using SVM$$+$$+ privileged information for manufacturing process modeling , 2017, Soft Comput..

[30]  Lixiang Li,et al.  The architecture of dynamic reservoir in the echo state network. , 2012, Chaos.

[31]  Tingwen Huang,et al.  Event-triggered H∞ state estimation for discrete-time neural networks with mixed time delays and sensor saturations , 2017, Neural Computing and Applications.