Statistical mechanics as the underlying theory of ‘elastic’ and ‘neural’ optimisations

There is an interesting connection between two, recently popular, methods for finding good approximate solutions to hard optimisation problems, the ‘neural’ approach of Hopfield and Tank and the elastic-net method of Durbin and Willshaw. They both have an underlying statistical mechanics foundation and can be derived as the leading approximation to the thermodynamic free energy of related physical models. The apparent difference in the form of the two algorithms comes from different handling of constraints when evaluating the thermodynamic partition function. If all the constraints are enforced ‘softly’, the ‘mean-field’ approximation to the thermodynamic free energy is just the neural network Lyapunov function. If, on the other hand, half of the constraints are enforced ‘strongly’, the leading approximation to the thermodynamic free energy is the elastic-net Lyapunov function. Our results have interesting implications for the general problem of mapping optimisation problems to ‘neural’ and ‘elastic’ netw...

[1]  D J Willshaw,et al.  A marker induction mechanism for the establishment of ordered neural mappings: its application to the retinotectal problem. , 1979, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[2]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[4]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[5]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[6]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[7]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[8]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[9]  Andrew Blake,et al.  Visual Reconstruction , 1987, Deep Learning for EEG-Based Brain–Computer Interfaces.

[10]  Carsten Peterson,et al.  A Mean Field Theory Learning Algorithm for Neural Networks , 1987, Complex Syst..

[11]  DAVID JOHNSON,et al.  More approaches to the travelling salesman guide , 1987, Nature.

[12]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[13]  M. Bertero,et al.  Ill-posed problems in early vision , 1988, Proc. IEEE.

[14]  Geoffrey C. Fox,et al.  The physical structure of concurrent problems and concurrent computers , 1988, Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences.

[15]  Marvin Minsky,et al.  Perceptrons: expanded edition , 1988 .

[16]  G. Fox,et al.  Neural Networks and Dynamic Complex Systems , 1989, [1989] Record of Proceedings. The 22nd Annual Simulation Symposium.