Learning by message-passing in networks of discrete synapses

We show that a message-passing process allows us to store in binary "material" synapses a number of random patterns which almost saturate the information theoretic bounds. We apply the learning algorithm to networks characterized by a wide range of different connection topologies and of size comparable with that of biological systems (e.g., [EQUATION: SEE TEXT]). The algorithm can be turned into an online-fault tolerant-learning protocol of potential interest in modeling aspects of synaptic plasticity and in building neuromorphic devices.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  J. F. Mansergh,et al.  Physique , 1903, The Indian medical gazette.

[3]  J. Thomson,et al.  Philosophical Magazine , 1945, Nature.

[4]  S. Kaplan.  The Physiology of Thought , 1950 .

[5]  L. Goddard.  Information Theory , 1962, Nature.

[6]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[7]  H. Kalmus.  Biological Cybernetics , 1972, Nature.

[8]  Franklin A. Graybill,et al.  Introduction to The theory , 1974 .

[9]  R. Palmer,et al.  Solution of 'Solvable model of a spin glass' , 1977 .

[10]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[11]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[12]  M. Mézard.  The space of interactions in neural networks: Gardner's computation with the cavity method , 1989 .

[13]  M. Tsodyks.  ASSOCIATIVE MEMORY IN NEURAL NETWORKS WITH BINARY SYNAPSES , 1990 .

[14]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[15]  I. Kanter,et al.  Storage Capacity of a Multilayer Neural Network with Binary Weights , 1991 .

[16]  Hansel,et al.  Broken symmetries in multilayered perceptrons. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[17]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[18]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[19]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[20]  H. Horner.  Dynamics of learning for the binary perceptron problem , 1992 .

[21]  D. Amit,et al.  Constraints on learning in dynamic synapses , 1992 .

[22]  R. W. Penney,et al.  The weight-space of the binary perceptron , 1993 .

[23]  Daniel J. Amit,et al.  Learning in Neural Networks with Material Synapses , 1994, Neural Computation.

[24]  Monasson,et al.  Analytical and numerical study of internal representations in multilayer neural networks with binary weights. , 1996, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[25]  Monasson,et al.  Entropy of the K-satisfiability problem. , 1996, Physical review letters.

[26]  A. Zador,et al.  Dynamic Synapses in the Cortex , 1997, Neuron.

[27]  Axthonv G. Oettinger,et al.  IEEE Transactions on Information Theory , 1998 .

[28]  J. Hopfield,et al.  All-or-none potentiation at CA3-CA1 synapses. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[29]  Jeong Han Kim,et al.  Covering Cubes by Random Half Cubes with Applications to Binary Neural Networks , 1998, J. Comput. Syst. Sci..

[30]  Christian Van den Broeck,et al.  Statistical Mechanics of Learning , 2001 .

[31]  D. Wilkin,et al.  Neuron , 2001, Brain Research.

[32]  Riccardo Zecchina,et al.  Coloring random graphs , 2002, Physical review letters.

[33]  M. Mézard,et al.  Random K-satisfiability problem: from an analytic solution to an efficient algorithm. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[34]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[35]  Terrence J Sejnowski,et al.  Communication in Neuronal Networks , 2003, Science.

[36]  R. Zecchina,et al.  Polynomial iterative algorithms for coloring and analyzing random graphs. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[37]  Y. Kabashima.  Propagating beliefs in spin-glass models , 2002, cond-mat/0211500.

[38]  Y. Kabashima.  A CDMA multiuser detection algorithm on the basis of belief propagation , 2003 .

[39]  J. Nadal,et al.  Optimal Information Storage and the Distribution of Synaptic Weights Perceptron versus Purkinje Cell , 2004, Neuron.

[40]  T. Murayama.  Thouless-Anderson-Palmer approach for lossy compression. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[41]  Konstantin K. Likharev,et al.  Neuromorphic architectures for nanoelectronic circuits , 2004, Int. J. Circuit Theory Appl..

[42]  R. Durbin,et al.  Bounds on the learning capacity of some multi-layer networks , 1989, Biological Cybernetics.

[43]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[44]  S. Wang,et al.  Graded bidirectional synaptic plasticity is composed of switch-like unitary events. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[45]  Yoshiyuki Kabashima,et al.  Statistical Mechanical Approach to Lossy Data Compression:Theory and Practice , 2005, ArXiv.

[46]  David Saad,et al.  Improved message passing for inference in densely connected systems , 2005, ArXiv.

[47]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[48]  M. Mézard,et al.  Survey-propagation decimation through distributed local computations , 2005, cond-mat/0512002.