Synergies between Intrinsic and Synaptic Plasticity Based on Information Theoretic Learning

In experimental and theoretical neuroscience, synaptic plasticity has dominated the area of neural plasticity for a very long time. Recently, neuronal intrinsic plasticity (IP) has become a hot topic in this area. IP is sometimes thought to be an information-maximization mechanism. However, it is still unclear how IP affects the performance of artificial neural networks in supervised learning applications. From an information-theoretical perspective, the error-entropy minimization (MEE) algorithm has newly been proposed as an efficient training method. In this study, we propose a synergistic learning algorithm combining the MEE algorithm as the synaptic plasticity rule and an information-maximization algorithm as the intrinsic plasticity rule. We consider both feedforward and recurrent neural networks and study the interactions between intrinsic and synaptic plasticity. Simulations indicate that the intrinsic plasticity rule can improve the performance of artificial neural networks trained by the MEE algorithm.

[1]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[2]  Niraj S. Desai,et al.  Plasticity in the intrinsic excitability of cortical pyramidal neurons , 1999, Nature Neuroscience.

[3]  Gordon Pipa,et al.  SORN: A Self-Organizing Recurrent Neural Network , 2009, Front. Comput. Neurosci..

[4]  Jochen Triesch,et al.  Synergies Between Intrinsic and Synaptic Plasticity Mechanisms , 2007, Neural Computation.

[5]  Ronald J. Williams,et al.  Gradient-based learning algorithms for recurrent networks and their computational complexity , 1995 .

[6]  D. Debanne,et al.  Long-term plasticity of intrinsic excitability: learning rules and mechanisms. , 2003, Learning & memory.

[7]  Robert H. Cudmore,et al.  Long-term potentiation of intrinsic excitability in LV visual cortical neurons. , 2004, Journal of neurophysiology.

[8]  Liang Li,et al.  Nonlinear adaptive prediction of nonstationary signals , 1995, IEEE Trans. Signal Process..

[9]  Danilo P. Mandic,et al.  Toward an optimal PRNN-based nonlinear predictor , 1999, IEEE Trans. Neural Networks.

[10]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[11]  Mei Zhang,et al.  Calcium signal-dependent plasticity of neuronal excitability developed postnatally. , 2004, Journal of neurobiology.

[12]  Niraj S. Desai,et al.  Homeostatic Plasticity and STDP: Keeping a Neuron's Cool in a Fluctuating World , 2010, Front. Syn. Neurosci..

[13]  E. Parzen.  On Estimation of a Probability Density Function and Mode , 1962 .

[14]  E Marder,et al.  Memory from the dynamics of intrinsic membrane currents. , 1996, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Deniz Erdoğmuş,et al.  COMPARISON OF ENTROPY AND MEAN SQUARE ERROR CRITERIA IN ADAPTIVE SYSTEM TRAINING USING HIGHER ORDER STATISTICS , 2004 .

[16]  Chunguang Li,et al.  A Spike-Based Model of Neuronal Intrinsic Plasticity , 2013, IEEE Transactions on Autonomous Mental Development.

[17]  Jose C. Principe,et al.  Information Theoretic Learning - Renyi's Entropy and Kernel Perspectives , 2010, Information Theoretic Learning.

[18]  Jochen Triesch,et al.  A Gradient Rule for the Plasticity of a Neuron's Intrinsic Excitability , 2005, ICANN.

[19]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[20]  Christof Koch,et al.  How voltage-dependent conductances can adapt to maximize the information encoded by neuronal firing rate , 1999, Nature Neuroscience.

[21]  Klaus Neumann,et al.  Batch Intrinsic Plasticity for Extreme Learning Machines , 2011, ICANN.

[22]  S. Laughlin.  A Simple Coding Procedure Enhances a Neuron's Information Capacity , 1981, Zeitschrift fur Naturforschung. Section C, Biosciences.

[23]  L. Abbott,et al.  Responses of neurons in primary and inferior temporal visual cortices to natural scenes , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[24]  Jochen J. Steil,et al.  Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning , 2007, Neural Networks.

[25]  D. Linden,et al.  The other side of the engram: experience-driven changes in neuronal intrinsic excitability , 2003, Nature Reviews Neuroscience.

[26]  Jonathon A. Chambers,et al.  Nonlinear adaptive prediction of speech with a pipelined recurrent neural network , 1998, IEEE Trans. Signal Process..

[27]  Chunguang Li,et al.  A Model of Neuronal Intrinsic Plasticity , 2011, IEEE Transactions on Autonomous Mental Development.

[28]  Jose C. Principe,et al.  Neural and Adaptive Systems: Fundamentals through Simulations with CD-ROM , 1999 .

[29]  J. Byrne,et al.  More than synaptic plasticity: role of nonsynaptic plasticity in learning and memory , 2010, Trends in Neurosciences.

[30]  Deniz Erdogmus,et al.  An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems , 2002, IEEE Trans. Signal Process..

[31]  Yves Chauvin,et al.  Backpropagation: theory, architectures, and applications , 1995 .