Randomized Recurrent Neural Networks

Neural Networks (NNs) with random weights represent nowadays a topic of consolidated use in the Machine Learning research community. In this contribution we focus in particular on recurrent NN models, which in a randomized setting represent a case of particular interest per se, entailing a number of intriguing research challenges primarily related to the control of the developed dynamics for the learning purposes. Framed in the Reservoir Computing paradigm, this paper aims at providing the basic elements and summarizing the recent advances on the topic of randomized recurrent NNs (divided in theoretical, architectural, deep learning and structured domain aspects), and introducing the papers accepted for the ESANN special session.

[1]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[2]  Claudio Gallicchio,et al.  Deep Echo State Networks for Diagnosis of Parkinson's Disease , 2018, ESANN.

[3]  Claudio Gallicchio,et al.  Architectural and Markovian factors of echo state networks , 2011, Neural Networks.

[4]  Claudio Gallicchio,et al.  Local Lyapunov exponents of deep echo state networks , 2018, Neurocomputing.

[5]  T. Tao.  Topics in Random Matrix Theory , 2012 .

[6]  Claudio Gallicchio,et al.  Deep Echo State Network (DeepESN): A Brief Survey , 2017, ArXiv.

[7]  Jean-Pierre Martens,et al.  Acoustic Modeling With Hierarchical Reservoirs , 2013, IEEE Transactions on Audio, Speech, and Language Processing.

[8]  Alessandro Sperduti,et al.  Challenges in Deep Learning , 2016, ESANN.

[9]  Junfei Qiao,et al.  Growing Echo-State Network With Multiple Subreservoirs , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[11]  Peter Tiño,et al.  Markovian architectural bias of recurrent neural networks , 2004, IEEE Transactions on Neural Networks.

[12]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[13]  Igor Farkas,et al.  Computational analysis of memory capacity in echo state networks , 2016, Neural Networks.

[14]  Lorenzo Livi,et al.  Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Haim Sompolinsky,et al.  Short-term memory in orthogonal neural networks. , 2004, Physical review letters.

[16]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[17]  Jochen Triesch,et al.  Synergies between Intrinsic and Synaptic Plasticity in Individual Model Neurons , 2004, NIPS.

[18]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[19]  Minoru Asada,et al.  Studies on reservoir initialization and dynamics shaping in echo state networks , 2009, ESANN.

[20]  Claudio Gallicchio,et al.  Short-term Memory of Deep RNN , 2018, ESANN.

[21]  Peter Tiño,et al.  Asymptotic Fisher memory of randomized linear symmetric Echo State Networks , 2018, Neurocomputing.

[22]  Peter Michael Young,et al.  A tighter bound for the echo state property , 2006, IEEE Transactions on Neural Networks.

[23]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[24]  Toshiyuki Yamane,et al.  Dynamics of Reservoir Computing at the Edge of Stability , 2016, ICONIP.

[25]  Alessio Micheli,et al.  Adaptive Contextual Processing of Structured Data by Recursive Neural Networks: A Survey of Computational Properties , 2007, Perspectives of Neural-Symbolic Integration.

[26]  Herbert Jaeger,et al.  Echo State Property Linked to an Input: Exploring a Fundamental Characteristic of Recurrent Neural Networks , 2013, Neural Computation.

[27]  Nils Bertschinger,et al.  Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks , 2004, Neural Computation.

[28]  Robert Jenssen,et al.  Bidirectional deep-readout echo state networks , 2017, ESANN.

[29]  Claudio Gallicchio,et al.  Deep reservoir computing: A critical experimental analysis , 2017, Neurocomputing.

[30]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[31]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[32]  Alessandro Sperduti,et al.  A general framework for adaptive processing of data structures , 1998, IEEE Trans. Neural Networks.

[33]  Claudio Gallicchio,et al.  Deep Reservoir Computing: A Critical Analysis , 2016, ESANN.

[34]  Claudio Gallicchio,et al.  Experimental Analysis of Deep Echo State Networks for Ambient Assisted Living , 2017, AI*AAL@AI*IA.

[35]  Benjamin Schrauwen,et al.  Memory versus non-linearity in reservoirs , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[36]  Dianhui Wang,et al.  Randomness in neural networks: an overview , 2017, WIREs Data Mining Knowl. Discov..

[37]  Minoru Asada,et al.  Information processing in echo state networks at the edge of chaos , 2011, Theory in Biosciences.

[38]  Peter Tiño,et al.  Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps , 2012, Neural Computation.

[39]  Peter Tiño,et al.  Recurrent Neural Networks with Small Weights Implement Definite Memory Machines , 2003, Neural Computation.

[40]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[41]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[42]  Simon Haykin,et al.  Decoupled echo state networks with lateral inhibition , 2007, Neural Networks.

[43]  Peter Tiño,et al.  Predictive Modeling with Echo State Networks , 2008, ICANN.

[44]  W. Maass,et al.  What makes a dynamical system computationally powerful ? , 2022 .

[45]  Claudio Gallicchio,et al.  Echo State Property of Deep Reservoir Computing Networks , 2017, Cognitive Computation.

[46]  Stefan J. Kiebel,et al.  Re-visiting the echo state property , 2012, Neural Networks.

[47]  Roger Labahn,et al.  Design Strategies for Weight Matrices of Echo State Networks , 2012, Neural Computation.

[48]  Sebastián Basterrech,et al.  Estimation of the Human Concentration using Echo State Networks , 2018, ESANN.

[49]  S. Massar,et al.  Mean Field Theory of Dynamical Systems Driven by External Signals , 2012, ArXiv.

[50]  Gilles Wainrib,et al.  A local Echo State Property through the largest Lyapunov exponent , 2016, Neural Networks.

[51]  Benjamin Schrauwen,et al.  On the Quantification of Dynamics in Reservoir Computing , 2009, ICANN.

[52]  Robert A. Legenstein,et al.  Methods for Estimating the Computational Power and Generalization Capability of Neural Microcircuits , 2004, NIPS.

[53]  Masanobu Inubushi,et al.  Reservoir Computing Beyond Memory-Nonlinearity Trade-off , 2017, Scientific Reports.

[54]  Claudio Gallicchio,et al.  Tree Echo State Networks , 2013, Neurocomputing.

[55]  Benjamin Schrauwen,et al.  On Computational Power and the Order-Chaos Phase Transition in Reservoir Computing , 2008, NIPS.

[56]  András Lörincz,et al.  Critical Echo State Networks , 2006, ICANN.

[57]  Claudio Gallicchio,et al.  Graph Echo State Networks , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[58]  Claudio Gallicchio,et al.  Randomized Machine Learning Approaches: Recent Developments and Challenges , 2017, ESANN.

[59]  Igor Farkas,et al.  Maximizing memory capacity of echo state networks with orthogonalized reservoirs , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[60]  Peter Tiño.  Fisher memory of linear Wigner echo state networks , 2017, ESANN.

[61]  Lorenzo Livi,et al.  Investigating echo state networks dynamics by means of recurrence analysis , 2016, IEEE Trans. Neural Networks Learn. Syst..

[62]  Benjamin Schrauwen,et al.  Phoneme Recognition with Large Hierarchical Reservoirs , 2010, NIPS.

[63]  M. C. Soriano,et al.  Advances in photonic reservoir computing , 2017 .

[64]  Hossam Faris,et al.  Forecasting Business Failure in Highly Imbalanced Distribution based on Delay Line Reservoir , 2018, ESANN.

[65]  Claudio Gallicchio,et al.  Local Lyapunov Exponents of Deep RNN , 2017, ESANN.

[66]  Peter Tiño,et al.  Predicting the Future of Discrete Sequences from Fractal Representations of the Past , 2001, Machine Learning.

[67]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[68]  Romain Couillet,et al.  The Asymptotic Performance of Linear Echo State Neural Networks , 2016, J. Mach. Learn. Res..

[69]  Claudio Gallicchio,et al.  Hierarchical Temporal Representation in Linear Reservoir Computing , 2017, Neural Advances in Processing Nonlinear Dynamic Signals.

[70]  Michael Kampffmeyer,et al.  Temporal overdrive recurrent neural network , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[71]  Robert A. Legenstein,et al.  2007 Special Issue: Edge of chaos and prediction of computational performance for neural circuit models , 2007 .

[72]  Benjamin Schrauwen,et al.  Training and Analysing Deep Recurrent Neural Networks , 2013, NIPS.

[73]  Alessandro Sperduti,et al.  Supervised neural networks for the classification of structures , 1997, IEEE Trans. Neural Networks.