Resource-efficient perceptron has sparse synaptic weight distribution

Resource-efficiency is important for biological func­tion of neurons. Using the perceptron as a model of a neuron, we show that resource-efficient learning implies sparse neural connectivity. The perceptron associates inputs to outputs by adjusting its synaptic weights. The learned synaptic weights are proposed to be the most resource-efficient by minimizing a biological resource cost given by the total absolute synaptic weight (l1-norm). Analytical methods from statistical physics and numerical simulations demonstrate that a resource-efficient perceptron has sparse connectivity. Sparseness decreases and resource usage increases with the number of associations to be learned. Our results have implications for synaptic connectivity in the cerebellum, where supervised learning is believed to happen.

[1]  D. Attwell,et al.  Updated Energy Budgets for Neural Computation in the Neocortex and Cerebellum , 2012, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[2]  Karel Svoboda,et al.  Rapid Functional Maturation of Nascent Dendritic Spines , 2009, Neuron.

[3]  Julio Chapeton,et al.  Efficient associative memory storage in cortical circuits of inhibitory and excitatory neurons , 2012, Proceedings of the National Academy of Sciences.

[4]  Sen Song,et al.  Highly Nonrandom Features of Synaptic Connectivity in Local Cortical Circuits , 2005, PLoS biology.

[5]  H. Jörntell,et al.  Parallel fibre receptive fields of Purkinje cells and interneurons are climbing fibre‐specific , 2001, The European journal of neuroscience.

[6]  Armen Stepanyants,et al.  Effects of homeostatic constraints on associative memory storage and synaptic connectivity of cortical circuits , 2015, Front. Comput. Neurosci..

[7]  Dmitri B Chklovskii,et al.  Synaptic Connectivity and Neuronal Morphology Two Sides of the Same Coin , 2004, Neuron.

[8]  Christian Van den Broeck,et al.  Statistical Mechanics of Learning , 2001 .

[9]  J. Nadal,et al.  Optimal Information Storage and the Distribution of Synaptic Weights Perceptron versus Purkinje Cell , 2004, Neuron.

[10]  Robert Tibshirani,et al.  1-norm Support Vector Machines , 2003, NIPS.

[11]  Dmitri B. Chklovskii,et al.  Synaptic Connectivity and Neuronal MorphologyTwo Sides of the Same Coin , 2004 .

[12]  H. Kasai,et al.  Structure–stability–function relationships of dendritic spines , 2003, Trends in Neurosciences.

[13]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[14]  Geoffrey A. Kerchner,et al.  Silent synapses and the emergence of a postsynaptic mechanism for LTP , 2009, Nature Reviews Neuroscience.

[15]  G. Mitchison.  Neuronal branching patterns and the economy of cortical wiring , 1991, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[16]  M. Mézard,et al.  Spin Glass Theory And Beyond: An Introduction To The Replica Method And Its Applications , 1986 .

[17]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[18]  B. Barbour,et al.  Properties of Unitary Granule Cell→Purkinje Cell Synapses in Adult Rat Cerebellar Slices , 2002, The Journal of Neuroscience.

[19]  松崎 政紀.  Dendritic spine geometry is critical for AMPA receptor expression in hippocampal CA1 pyramidal neurons , 2001 .

[20]  Nicolas Brunel,et al.  Is cortical connectivity optimized for storing information? , 2016, Nature Neuroscience.

[21]  Lav R. Varshney,et al.  Optimal Information Storage in Noisy Synapses under Resource Constraints , 2006, Neuron.

[22]  D. Marr.  A theory of cerebellar cortex , 1969, The Journal of physiology.

[23]  G. Buzsáki,et al.  Inhibition and Brain Work , 2007, Neuron.

[24]  S. Laughlin,et al.  An Energy Budget for Signaling in the Grey Matter of the Brain , 2001, Journal of cerebral blood flow and metabolism : official journal of the International Society of Cerebral Blood Flow and Metabolism.

[25]  D J Amit,et al.  The interaction space of neural networks with sign-constrained synapses , 1989 .

[26]  Rob R. de Ruyter van Steveninck,et al.  The metabolic cost of neural information , 1998, Nature Neuroscience.

[27]  John Porrill,et al.  Silent Synapses, LTP, and the Indirect Parallel-Fibre Pathway: Computational Consequences of Optimal Cerebellar Noise-Processing , 2008, PLoS Comput. Biol..

[28]  Mark C. W. van Rossum,et al.  Energy Efficient Sparse Connectivity from Imbalanced Synaptic Plasticity Rules , 2015, PLoS Comput. Biol..

[29]  E. Gardner.  The space of interactions in neural network models , 1988 .

[30]  William B. Levy,et al.  Energy Efficient Neural Codes , 1996, Neural Computation.

[31]  J. Albus.  A Theory of Cerebellar Function , 1971 .