Hierarchical neural networks for multivariate time series prediction

Considering the problem that for multivariate time series prediction, the adaptation of a single reservoir may not be sufficient to improve the prediction accuracy, we propose a novel hierarchical neural network herein. In the first hierarchy, several simplified echo state networks - simple cycle reservoirs (SCRs) are used to extract the dynamical features of the multivariate time series. Particle swarm optimization method is conducted in the pre-training stage to optimize the free parameters of SCRs. The reservoir states of SCRs are collected as dynamical features. In the second hierarchy, a feature selection method based on mutual information is used to select a compact feature set as the input for the extreme learning machine (ELM). In order to further improve the prediction accuracy, the optimal number of hidden nodes of the ELM is chosen by a modified recursive algorithm. Simulation results on monthly average temperature and rainfall series in Dalian China sustain that the proposed model is effective for multivariate time series.

[1]  Jane Labadin,et al.  Feature selection based on mutual information , 2015, 2015 9th International Conference on IT in Asia (CITA).

[2]  Akinori Nishihara,et al.  Evolutionary pre-training for CRJ-type reservoir of echo state networks , 2015, Neurocomputing.

[3]  Fernando José Von Zuben,et al.  An extended echo state network using Volterra filtering and principal component analysis , 2012, Neural Networks.

[4]  Chih-Min Lin,et al.  An Efficient Interval Type-2 Fuzzy CMAC for Chaos Time-Series Prediction and Synchronization , 2014, IEEE Transactions on Cybernetics.

[5]  Yiannis Demiris,et al.  Spatio-Temporal Learning With the Online Finite and Infinite Echo-State Gaussian Processes , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[6]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[7]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Yiannis Demiris,et al.  The copula echo state network , 2012, Pattern Recognit..

[9]  Yue Joseph Wang,et al.  Nonlinear System Modeling With Random Matrices: Echo State Networks Revisited , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Fionn Murtagh,et al.  Ultrametric Wavelet Regression of Multivariate Time Series: Application to Colombian Conflict Analysis , 2009, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[11]  Guang-Bin Huang,et al.  An Insight into Extreme Learning Machines: Random Neurons, Random Features and Kernels , 2014, Cognitive Computation.

[12]  Heiko Wersing,et al.  Evolutionary optimization of a hierarchical object recognition model , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Henry Leung,et al.  Prediction of noisy chaotic time series using an optimal radial basis function neural network , 2001, IEEE Trans. Neural Networks.

[14]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[15]  Min Han,et al.  A modified fast recursive hidden nodes selection algorithm for ELM , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[16]  Cyrus Shahabi,et al.  Feature subset selection and feature ranking for multivariate time series , 2005, IEEE Transactions on Knowledge and Data Engineering.

[17]  Jang Myung Lee,et al.  Precise Positioning of Nonsmooth Dynamic Systems Using Fuzzy Wavelet Echo State Networks and Dynamic Surface Sliding Mode Control , 2013, IEEE Transactions on Industrial Electronics.

[18]  Benjamin Schrauwen,et al.  Recurrent Kernel Machines: Computing with Infinite Echo State Networks , 2012, Neural Computation.

[19]  Tingting Wang,et al.  Recovering Chaotic Properties From Small Data , 2014, IEEE Transactions on Cybernetics.

[20]  Danilo P. Mandic,et al.  An Augmented Echo State Network for Nonlinear Adaptive Filtering of Complex Noncircular Signals , 2011, IEEE Transactions on Neural Networks.

[21]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[22]  J. Koenderink.  Q… , 2014, Les noms officiels des communes de Wallonie, de Bruxelles-Capitale et de la communaute germanophone.

[23]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[24]  Min Han,et al.  Support Vector Echo-State Machine for Chaotic Time-Series Prediction , 2007, IEEE Transactions on Neural Networks.

[25]  Jan Danckaert,et al.  Delay-Based Reservoir Computing: Noise Effects in a Combined Analog and Digital Implementation , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Xiaohui Liu,et al.  Variable grouping in multivariate time series via correlation , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[27]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[28]  Ravi Sankar,et al.  Time Series Prediction Using Support Vector Machines: A Survey , 2009, IEEE Computational Intelligence Magazine.

[29]  Jochen J. Steil,et al.  Reservoir regularization stabilizes learning of Echo State Networks with output feedback , 2011, ESANN.

[30]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[31]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[32]  Benjamin Schrauwen,et al.  A hierarchy of recurrent networks for speech recognition , 2009, NIPS 2009.

[33]  Guillaume Patry,et al.  Constructing a model hierarchy with background knowledge for structural risk minimization: application to biological treatment of wastewater , 2006, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.