Stationary log-normal distribution of weights stems from spontaneous ordering in adaptive node networks

Experimental evidence recently indicated that neural networks can learn in a different manner than was previously assumed, using adaptive nodes instead of adaptive links. Consequently, links to a node undergo the same adaptation, resulting in cooperative nonlinear dynamics with oscillating effective link weights. Here we show that the biological reality of stationary log-normal distribution of effective link weights in neural networks is a result of such adaptive nodes, although each effective link weight varies significantly in time. The underlying mechanism is a stochastic restoring force emerging from a spontaneous temporal ordering of spike pairs, generated by strong effective link preceding by a weak one. In addition, for feedforward adaptive node networks the number of dynamical attractors can scale exponentially with the number of links. These results are expected to advance deep learning capabilities and to open horizons to an interplay between adaptive node rules and the distribution of network link weights.

[1]  Ido Kanter,et al.  Adaptive nodes enrich nonlinear cooperative learning beyond traditional adaptation by links , 2018, Scientific Reports.

[2]  N. Spruston.  Pyramidal neurons: dendritic structure and synaptic integration , 2008, Nature Reviews Neuroscience.

[3]  Youngjin Park,et al.  Symmetry of learning rate in synaptic plasticity modulates formation of flexible and stable memories , 2017, Scientific Reports.

[4]  Shivayogi M Hugar,et al.  An In Vivo Study , 2015 .

[5]  Federico Levi,et al.  The discovery of skewness , 2018, Nature Physics.

[6]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[7]  Lenka Zdeborová,et al.  New tool in the box , 2017, Nature Physics.

[8]  Amir Bashan,et al.  Network physiology reveals relations between network topology and physiological function , 2012, Nature Communications.

[9]  G. Laurent,et al.  Conditional modulation of spike-timing-dependent plasticity for olfactory learning , 2012, Nature.

[10]  Sen Song,et al.  Highly Nonrandom Features of Synaptic Connectivity in Local Cortical Circuits , 2005, PLoS biology.

[11]  Ido Kanter,et al.  Synchronization among neuronal pools without common inputs: in vivo study , 2014, Brain Structure and Function.

[12]  Y. Dan,et al.  Spike timing-dependent plasticity: from synapse to perception. , 2006, Physiological reviews.

[13]  Gerardo Iñiguez,et al.  Threshold driven contagion on weighted networks , 2017, Scientific Reports.

[14]  Bo Li,et al.  Exploring the Function Space of Deep-Learning Machines , 2017, Physical review letters.

[15]  Christian Van den Broeck,et al.  Statistical Mechanics of Learning , 2001 .

[16]  D. R. Muir,et al.  Functional organization of excitatory synaptic strength in primary visual cortex , 2015, Nature.

[17]  Ido Kanter,et al.  Neuronal response impedance mechanism implementing cooperative networks with low firing rates and μs precision , 2014, Front. Neural Circuits.

[18]  Ido Kanter,et al.  New Types of Experiments Reveal that a Neuron Functions as Multiple Independent Threshold Units , 2017, Scientific Reports.

[19]  Wulfram Gerstner,et al.  Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. , 2005, Journal of neurophysiology.

[20]  Y. Loewenstein,et al.  Multiplicative Dynamics Underlie the Emergence of the Log-Normal Distribution of Spine Sizes in the Neocortex In Vivo , 2011, The Journal of Neuroscience.

[21]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[22]  Moshe Abeles,et al.  Corticonics: Neural Circuits of Cerebral Cortex , 1991 .

[23]  Rosario N. Mantegna,et al.  Plasticity of brain wave network interactions and evolution across physiologic states , 2015, Front. Neural Circuits.

[24]  Y.-Y. Liu,et al.  The fundamental advantages of temporal networks , 2016, Science.

[25]  Lucas C Parra,et al.  Finding influential nodes for integration in brain networks using optimal percolation theory , 2018, Nature Communications.

[26]  M. Opper.  Learning in Neural Networks: Solvable Dynamics , 1989 .

[27]  Yee Lian Chew,et al.  Network control principles predict neuron function in the Caenorhabditis elegans connectome , 2017, Nature.

[28]  Jacob G. Scott,et al.  Evolutionary dynamics of incubation periods , 2017, bioRxiv.

[29]  G. Buzsáki,et al.  The log-dynamic brain: how skewed distributions affect network operations , 2014, Nature Reviews Neuroscience.

[30]  Mark Buchanan.  Depths of learning , 2015 .

[31]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[32]  Zoubin Ghahramani,et al.  Probabilistic machine learning and artificial intelligence , 2015, Nature.