Learning may need only a few bits of synaptic precision.

Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning and experiments, and possibly by hardware implementation considerations as well. In this paper we extend a previous large deviations analysis which unveiled the existence of peculiar dense regions in the space of synaptic states which accounts for the possibility of learning efficiently in networks with binary synapses. We extend the analysis to synapses with multiple states and generally more plausible biological features. The results clearly indicate that the overall qualitative picture is unchanged with respect to the binary case, and very robust to variation of the details of the model. We also provide quantitative results which suggest that the advantages of increasing the synaptic precision (i.e., the number of internal synaptic states) rapidly vanish after the first few bits, and therefore that, for practical applications, only few bits may be needed for near-optimal performance, consistent with recent biological findings. Finally, we demonstrate how the theoretical analysis can be exploited to design efficient algorithmic search strategies.

[1]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[2]  E. Gardner,et al.  Three unfinished works on the optimal storage capacity of networks , 1989 .

[3]  H. Gutfreund,et al.  Capacity of neural networks with discrete synaptic couplings , 1990 .

[4]  Sompolinsky,et al.  Learning from examples in large neural networks. , 1990, Physical review letters.

[5]  H. Horner.  Dynamics of learning for the binary perceptron problem , 1992 .

[6]  G. Parisi,et al.  Recipes for metastable states in spin glasses , 1995 .

[7]  Jeong Han Kim,et al.  Covering Cubes by Random Half Cubes with Applications to Binary Neural Networks , 1998, J. Comput. Syst. Sci..

[8]  Michel Talagrand.  Intersecting random half cubes , 1999, Random Struct. Algorithms.

[9]  J. Nadal,et al.  Optimal Information Storage and the Distribution of Synaptic Weights Perceptron versus Purkinje Cell , 2004, Neuron.

[10]  O. Rivoire.  Properties of Atypical Graphs from Negative Complexities , 2003, cond-mat/0312501.

[11]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[12]  S. Wang,et al.  Graded bidirectional synaptic plasticity is composed of switch-like unitary events. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[13]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[14]  Riccardo Zecchina,et al.  Learning by message-passing in networks of discrete synapses , 2005, Physical review letters.

[15]  Nicolas Brunel,et al.  Efficient supervised learning in networks with binary synapses , 2007, BMC Neuroscience.

[16]  Andrea Montanari,et al.  Gibbs states and the set of solutions of random constraint satisfaction problems , 2006, Proceedings of the National Academy of Sciences.

[17]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[18]  Carlo Baldassi.  Generalization Learning in a Perceptron with Binary Synapses , 2009, 1211.3024.

[19]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[20]  Yoshiyuki Kabashima,et al.  Origin of the computational hardness for learning with binary synapses , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[21]  Carlo Baldassi,et al.  Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. , 2015, Physical review letters.

[22]  Carlo Baldassi,et al.  Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems , 2015 .

[23]  T. Sejnowski,et al.  Hippocampal Spine Head Sizes Are Highly Precise , 2015, bioRxiv.

[24]  Carlo Baldassi,et al.  A Max-Sum algorithm for training discrete neural networks , 2015, ArXiv.