ECHO STATE QUEUEING NETWORKS: A COMBINATION OF RESERVOIR COMPUTING AND RANDOM NEURAL NETWORKS

This paper deals with two ideas appeared during the last developing phase in Artificial Intelligence: Reservoir Computing (RC) and Random Neural Networks. Both have been very successful in many applications. We propose a new model belonging to the first class, taking the structure of the second for its dynamics. The new model is called Echo State Queuing Network. The paper positions the model in the global Machine Learning area, and provides examples of its use and performances. We show on largely used benchmarks that it is a very accurate tool, and we illustrate how it compares with standard RC models.

[1]  Erol Gelenbe,et al.  Deep learning with random neural networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[2]  Gerardo Rubino,et al.  A study of real-time packet video quality using random neural networks , 2002, IEEE Trans. Circuits Syst. Video Technol..

[3]  Gerardo Rubino,et al.  Levenberg - Marquardt Training Algorithms for Random Neural Networks , 2011, Comput. J..

[4]  Michael I. Jordan.  Serial Order: A Parallel Distributed Processing Approach , 1997 .

[5]  Gerhard Tutz,et al.  Boosting techniques for nonlinear time series models , 2012 .

[6]  Kenneth L. Kraemer,et al.  One Laptop per Child: Vision vs. Reality , 2022 .

[7]  Erol Gelenbe,et al.  Random Neural Networks with Negative and Positive Signals and Product Form Solution , 1989, Neural Computation.

[8]  Mantas Lukoševičius,et al.  On self-organizing reservoirs and their hierarchies , 2010 .

[9]  Taskin Koçak,et al.  Survey of random neural network applications , 2000, Eur. J. Oper. Res..

[10]  Gabriel Alfredo Baum,et al.  A new approach for the prediction of end-to-end performance of multimedia streams , 2004 .

[11]  Erol Gelenbe,et al.  Random Neural Networks with Synchronized Interactions , 2008, Neural Computation.

[12]  E. Gelenbe.  Product-form queueing networks with negative and positive customers , 1991 .

[13]  Eduardo Sontag,et al.  Turing computability with neural nets , 1991 .

[14]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[15]  Vaclav Snasel,et al.  A Performance Study of Random Neural Network as Supervised Learning Tool Using CUDA , 2016 .

[16]  Erol Gelenbe,et al.  Big Data for Autonomic Intercontinental Overlays , 2016, IEEE Journal on Selected Areas in Communications.

[17]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[18]  Erol Gelenbe,et al.  G-Networks with resets , 2002 .

[19]  Erol Gelenbe,et al.  Steps toward self-aware networks , 2009, CACM.

[20]  Henry Leung,et al.  Prediction Intervals for a Noisy Nonlinear Time Series Based on a Bootstrapping Reservoir Computing Network Ensemble , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Gerardo Rubino,et al.  Echo State Queueing Network: A new reservoir computing learning tool , 2012, 2013 IEEE 10th Consumer Communications and Networking Conference (CCNC).

[22]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[23]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[24]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[25]  Enrique Alba,et al.  An experimental analysis of the Echo State Network initialization using the Particle Swarm Optimization , 2014, 2014 Sixth World Congress on Nature and Biologically Inspired Computing (NaBIC 2014).

[26]  R. Temam,et al.  The incremental unknown method II , 1991 .

[27]  Andreas Stafylopatis,et al.  Training the random neural network using quasi-Newton methods , 2000, Eur. J. Oper. Res..

[28]  B. Schrauwen,et al.  Reservoir computing and extreme learning machines for non-linear time-series data analysis , 2013, Neural Networks.

[29]  Erol Gelenbe,et al.  Learning in the multiple class random neural network , 2002, IEEE Trans. Neural Networks.

[30]  Sebastián Basterrech,et al.  Empirical analysis of the necessary and sufficient conditions of the echo state property , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[31]  J.J. Steil,et al.  Backpropagation-decorrelation: online recurrent learning with O(N) complexity , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[32]  S. Basterrech,et al.  Random neural network model for supervised learning problems , 2015 .

[33]  Gilles Wainrib,et al.  A local Echo State Property through the largest Lyapunov exponent , 2016, Neural Networks.

[34]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[35]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[36]  Erol Gelenbe,et al.  The spiked random neural network: nonlinearity, learning and approximation , 1998, 1998 Fifth IEEE International Workshop on Cellular Neural Networks and their Applications. Proceedings (Cat. No.98TH8359).

[37]  Stelios Timotheou,et al.  The Random Neural Network: A Survey , 2010, Comput. J..

[38]  Gerardo Rubino,et al.  A Tutorial about Random Neural Networks in Supervised Learning , 2016, ArXiv.

[39]  Erol Gelenbe,et al.  Stability of Product Form G-Networks , 1992 .

[40]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[41]  Wolfgang Maass,et al.  Noisy Spiking Neurons with Temporal Coding have more Computational Power than Sigmoidal Neurons , 1996, NIPS.

[42]  Teresa Bernarda Ludermir,et al.  An approach to reservoir computing design and training , 2013, Expert Syst. Appl..

[43]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[44]  Erol Gelenbe,et al.  Learning in the Recurrent Random Neural Network , 1992, Neural Computation.

[45]  Erol Gelenbe.  G-Networks with Signals and Batch Removal , 1993 .

[46]  José Carlos Príncipe,et al.  Special issue on echo state networks and liquid state machines , 2007, Neural Networks.

[47]  Gerardo Rubino,et al.  Self-Organizing Maps and Scale-Invariant Maps in Echo State Networks , 2011, 2011 11th International Conference on Intelligent Systems Design and Applications.

[48]  Zhi-Hong Mao,et al.  Function approximation with spiked random networks , 1999, IEEE Trans. Neural Networks.