Forward and backward input variable selection for polynomial echo state networks

Abstract As extension of traditional echo state networks (ESNs), the polynomial echo state networks (PESNs) have been proposed in our previous work (C.L. Yang, J.F. Qiao, H.G. Han, L. Wang, Design of polynomial echo state networks for time series prediction, Neurocomputing 290 (2018) 148-160.) by employing the polynomial function of complete input variable as output weight matrix. In practice, the generalization performance and computational burden of PESNs are perturbed by redundant or irrelevant inputs. To construct output weights with a suitable subset of input variables, the forward selection based PESN (FS-PESN) and backward selection based PESN (BS-PESN) are proposed. Firstly, the forward selection method is used in FS-PESN to choose the input variable which incurs the maximum reduction on objective function, and the backward selection shame is introduced in BS-PESN to remove the input variable which leads to the smallest increment on objective function. Then, the iterative updating strategies are designed to avoid repetitive computations in FS-PESN and BS-PESN. Specially, an accelerating scheme is introduced into BS-PESN to simplify training process. Finally, numerical simulations are carried out to illustrate effectiveness of the proposed techniques in terms of generalization ability and testing time.

[1]  Xinguang He,et al.  A hybrid wavelet neural network model with mutual information and particle swarm optimization for forecasting monthly rainfall , 2015 .

[2]  Lin Lin,et al.  Genetic algorithm optimized double-reservoir echo state network for multi-regime time series prediction , 2017, Neurocomputing.

[3]  Xiaohua Wang,et al.  Echo State Networks With Orthogonal Pigeon-Inspired Optimization for Image Restoration , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Hendrik Van Brussel,et al.  Pruning and regularization in reservoir computing , 2009, Neurocomputing.

[5]  Nadia Nedjah,et al.  Online phoneme recognition using multi-layer perceptron networks combined with recurrent non-linear autoregressive neural networks with exogenous inputs , 2017, Neurocomputing.

[6]  N. Levinson.  The Wiener (Root Mean Square) Error Criterion in Filter Design and Prediction , 1946 .

[7]  Álvaro Alonso,et al.  Regression tree ensembles for wind energy and solar radiation prediction , 2017, Neurocomputing.

[8]  Fu Li,et al.  A pruning method of refining recursive reduced least squares support vector regression , 2015, Inf. Sci..

[9]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[10]  Joel A. Tropp,et al.  Signal Recovery From Random Measurements Via Orthogonal Matching Pursuit , 2007, IEEE Transactions on Information Theory.

[11]  Xuefeng Yan,et al.  Optimizing the echo state network with a binary particle swarm optimization algorithm , 2015, Knowl. Based Syst..

[12]  Stephen A. Billings,et al.  Ultra-Orthogonal Forward Regression Algorithms for the Identification of Non-Linear Dynamic Systems , 2016, Neurocomputing.

[13]  Holger R. Maier,et al.  Review of Input Variable Selection Methods for Artificial Neural Networks , 2011 .

[14]  Junfei Qiao,et al.  Online sequential echo state network with sparse RLS algorithm for time series prediction , 2019, Neural Networks.

[15]  Robert Shorten,et al.  A Note on Recursive Schur Complements, Block Hurwitz Stability of Metzler Matrices, and Related Results , 2017, IEEE Transactions on Automatic Control.

[16]  Min Han,et al.  Adaptive Elastic Echo State Network for Multivariate Time Series Prediction , 2016, IEEE Transactions on Cybernetics.

[17]  Lei Wang,et al.  Design of polynomial echo state networks for time series prediction , 2018, Neurocomputing.

[18]  Junfei Qiao,et al.  Constructive algorithm for fully connected cascade feedforward neural networks , 2016, Neurocomputing.

[19]  Tomaz Curk,et al.  Approximate multiple kernel learning with least-angle regression , 2019, Neurocomputing.

[20]  Andrew Chi-Sing Leung,et al.  Orthogonal least squares based center selection for fault-tolerant RBF networks , 2019, Neurocomputing.

[21]  Holger R. Maier,et al.  Non-linear variable selection for artificial neural networks using partial mutual information , 2008, Environ. Model. Softw..

[22]  Abdelkader Benyettou,et al.  Novel Approach Using Echo State Networks for Microscopic Cellular Image Segmentation , 2016, Cognitive Computation.

[23]  J. L. Hock,et al.  An exact recursion for the composite nearest‐neighbor degeneracy for a 2×N lattice space , 1984 .

[24]  Gaël Varoquaux,et al.  Recursive Nearest Agglomeration (ReNA): Fast Clustering for Approximation of Structured Signals , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Yong-Ping Zhao,et al.  Feature selection of generalized extreme learning machine for regression problems , 2018, Neurocomputing.

[26]  Dianhui Wang,et al.  A decentralized training algorithm for Echo State Networks in distributed big data applications , 2016, Neural Networks.

[27]  Pascal Vincent,et al.  Kernel Matching Pursuit , 2002, Machine Learning.

[28]  Jian Yang,et al.  A new recurrent neural network with noise-tolerance and finite-time convergence for dynamic quadratic minimization , 2018, Neurocomputing.

[29]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[30]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[31]  Daeyoung Kim,et al.  A Novel Echo State Network Model Using Bayesian Ridge Regression and Independent Component Analysis , 2018, ICANN.

[32]  Pilsung Kang,et al.  Locally linear ensemble for regression , 2018, Inf. Sci..

[33]  Tianyou Chai,et al.  Modeling collinear data using double-layer GA-based selective ensemble kernel partial least squares algorithm , 2017, Neurocomputing.

[34]  Jun Wang,et al.  Chaotic Time Series Prediction Based on a Novel Robust Echo State Network , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Gene H. Golub,et al.  Matrix computations , 1983 .