Constraint satisfaction problems and neural networks: A statistical physics perspective

A new field of research is rapidly expanding at the crossroad between statistical physics, information theory and combinatorial optimization. In particular, the use of cutting edge statistical physics concepts and methods allow one to solve very large constraint satisfaction problems like random satisfiability, coloring, or error correction. Several aspects of these developments should be relevant for the understanding of functional complexity in neural networks. On the one hand the message passing procedures which are used in these new algorithms are based on local exchange of information, and succeed in solving some of the hardest computational problems. On the other hand some crucial inference problems in neurobiology, like those generated in multi-electrode recordings, naturally translate into hard constraint satisfaction problems. This paper gives a non-technical introduction to this field, emphasizing the main ideas at work in message passing strategies and their possible relevance to neural networks modelling. It also introduces a new message passing algorithm for inferring interactions between variables from correlation data, which could be useful in the analysis of multi-electrode recording data.

[1]  W. P. Russ,et al.  Evolutionary information for specifying a protein fold , 2005, Nature.

[2]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[3]  S. Wang,et al.  Graded bidirectional synaptic plasticity is composed of switch-like unitary events. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[4]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[5]  Andrew Zisserman,et al.  Advances in Neural Information Processing Systems (NIPS) , 2007 .

[6]  Andrea Montanari,et al.  Modern Coding Theory: The Statistical Mechanics and Computer Science Point of View , 2007, ArXiv.

[7]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[8]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[9]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[10]  Gašper Tkačik,et al.  Information flow in biological networks , 2007 .

[11]  Michael J. Berry,et al.  Weak pairwise correlations imply strongly correlated network states in a neural population , 2005, Nature.

[12]  Riccardo Zecchina,et al.  Learning by message-passing in networks of discrete synapses , 2005, Physical review letters.

[13]  W. Bialek,et al.  Statistical mechanics of letters in words. , 2010, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  Rémi Monasson,et al.  Determining computational complexity from characteristic ‘phase transitions’ , 1999, Nature.

[15]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[16]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[17]  Stephen A. Cook,et al.  The complexity of theorem-proving procedures , 1971, STOC.

[18]  J. Hopfield,et al.  All-or-none potentiation at CA3-CA1 synapses. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[19]  Hilbert J. Kappen,et al.  Efficient Learning in Boltzmann Machines Using Linear Response Theory , 1998, Neural Computation.

[20]  W. P. Russ,et al.  Natural-like function in artificial WW domains , 2005, Nature.

[21]  Robert G. Gallager,et al.  Low-density parity-check codes , 1962, IRE Trans. Inf. Theory.

[22]  Rüdiger L. Urbanke,et al.  Modern Coding Theory , 2008 .

[23]  M. Mézard,et al.  Random K-satisfiability problem: from an analytic solution to an efficient algorithm. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[24]  S Kirkpatrick,et al.  Critical Behavior in the Satisfiability of Random Boolean Expressions , 1994, Science.

[25]  Bart Selman,et al.  Critical Behavior in the Computational Cost of Satisfiability Testing , 1996, Artif. Intell..

[26]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[27]  E. Friedgut,et al.  Sharp thresholds of graph properties, and the -sat problem , 1999 .

[28]  Michael J. Berry,et al.  Network information and connected correlations. , 2003, Physical review letters.

[29]  Greg J. Stephens,et al.  Toward a statistical mechanics of four letter words , 2007, ArXiv.

[30]  S. Kak.  Information, physics, and computation , 1996 .

[31]  Brendan J. Frey,et al.  Factor graphs and the sum-product algorithm , 2001, IEEE Trans. Inf. Theory.

[32]  Robert E. Schapire,et al.  Faster solutions of the inverse pairwise Ising problem , 2008 .

[33]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .