Echo state networks regulated by local intrinsic plasticity rules for regression

Abstract Intrinsic plasticity, as a biologically inspired unsupervised learning rule, is used for adapting the intrinsic excitability of the reservoir neurons. Existing intrinsic plasticity rules can only select a set of fixed rule parameters for the whole reservoir neurons, which affects the learning performance due to the lack of flexibility in providing intrinsic plasticity. In this paper, we present an echo state network (ESN) with local intrinsic plasticity rule built by different reservoir neurons which can adopt the intrinsic plasticity rule with different rule parameters to adjust its intrinsic excitability. And the covariance matrix adaptation evolution strategy is used to search and select the rule parameters corresponding to different reservoir neurons. Compared with several state-of-the-art ESN models and an ESN with the global plasticity rule, the proposed local intrinsic plasticity rule is able to achieve much better performance in some benchmark prediction tasks.

[1]  Gauss M. Cordeiro,et al.  The beta modified Weibull distribution , 2010, Lifetime data analysis.

[2]  Nikolaus Hansen,et al.  The CMA Evolution Strategy: A Tutorial , 2016, ArXiv.

[3]  Christof Koch,et al.  How voltage-dependent conductances can adapt to maximize the information encoded by neuronal firing rate , 1999, Nature Neuroscience.

[4]  Jochen Triesch,et al.  Synergies Between Intrinsic and Synaptic Plasticity Mechanisms , 2007, Neural Computation.

[5]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[6]  Jiri Pospichal,et al.  Improving the Prediction Accuracy of Echo State Neural Networks by Anti-Oja's Learning , 2007, ICANN.

[7]  Zhidong Deng,et al.  Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution , 2007, IEEE Transactions on Neural Networks.

[8]  Lei Wang,et al.  Design of polynomial echo state networks for time series prediction , 2018, Neurocomputing.

[9]  Jochen Triesch,et al.  A Gradient Rule for the Plasticity of a Neuron's Intrinsic Excitability , 2005, ICANN.

[10]  Adel M. Alimi,et al.  Optimized Echo State Network with Intrinsic Plasticity for EEG-Based Emotion Recognition , 2017, ICONIP.

[11]  Lin Lin,et al.  Genetic algorithm optimized double-reservoir echo state network for multi-regime time series prediction , 2017, Neurocomputing.

[12]  Yaochu Jin,et al.  Modeling neural plasticity in echo state networks for classification and regression , 2016, Inf. Sci..

[13]  Qian Li,et al.  Reservoir Computing with Both Neuronal Intrinsic Plasticity and Multi-Clustered Structure , 2017, Cognitive Computation.

[14]  W. Weibull.  A Statistical Distribution Function of Wide Applicability , 1951 .

[15]  Jun Wang,et al.  A Multi-objective Optimization Model for Alloy Addition in BOS Process Based on ESN and Modified MOPSO , 2015, 2015 Second International Conference on Mathematics and Computers in Sciences and in Industry (MCSI).

[16]  William B. Levy,et al.  Energy Efficient Neural Codes , 1996, Neural Computation.

[17]  M. Hénon,et al.  A two-dimensional mapping with a strange attractor , 1976 .

[18]  Min Han,et al.  Laplacian Echo State Network for Multivariate Time Series Prediction , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[19]  Chunguang Li,et al.  Synergies between Intrinsic and Synaptic Plasticity Based on Information Theoretic Learning , 2013, PloS one.

[20]  H. Jaeger,et al.  Reservoir riddles: suggestions for echo state network research , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[21]  Hui Liu,et al.  Colour image encryption based on advanced encryption standard algorithm with two-dimensional chaotic map , 2013, IET Inf. Secur..

[22]  Bernhard Sendhoff,et al.  Evolutionary Multi-objective Optimization of Spiking Neural Networks , 2007, ICANN.

[23]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[24]  L. Abbott,et al.  Responses of neurons in primary and inferior temporal visual cortices to natural scenes , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[25]  P. Földiák,et al.  Forming sparse representations by local anti-Hebbian learning , 1990, Biological Cybernetics.

[26]  Zhidong Deng,et al.  Synchronization enhancement of a scale-free highly clustered network , 2008, 2008 7th World Congress on Intelligent Control and Automation.

[27]  Yaochu Jin,et al.  Computational modeling of neural plasticity for self-organization of neural networks , 2014, Biosyst..

[28]  Herbert Jaeger,et al.  Optimization and applications of echo state networks with leaky- integrator neurons , 2007, Neural Networks.

[29]  Chunguang Li,et al.  A Model of Neuronal Intrinsic Plasticity , 2011, IEEE Transactions on Autonomous Mental Development.

[30]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[31]  Adel M. Alimi,et al.  Single- and multi-objective particle swarm optimization of reservoir structure in Echo State Network , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[32]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[33]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[34]  D. Linden,et al.  The other side of the engram: experience-driven changes in neuronal intrinsic excitability , 2003, Nature Reviews Neuroscience.

[35]  Jochen J. Steil,et al.  Intrinsic plasticity for reservoir learning algorithms , 2007, ESANN.

[36]  Junfei Qiao,et al.  Growing Echo-State Network With Multiple Subreservoirs , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[37]  Tie Qiu,et al.  Interval Type-2 Fuzzy Neural Networks for Chaotic Time Series Prediction: A Concise Overview , 2019, IEEE Transactions on Cybernetics.

[38]  Xavier Hinaut,et al.  Activity recognition with echo state networks using 3D body joints and objects category , 2016, ESANN.

[39]  Maciej Ogorzalek,et al.  Identification of chaotic systems based on adaptive synchronization , 1997 .