Statistical Physics Algorithms That Converge

In recent years there has been significant interest in adapting techniques from statistical physics, in particular mean field theory, to provide deterministic heuristic algorithms for obtaining approximate solutions to optimization problems. Although these algorithms have been shown experimentally to be successful there has been little theoretical analysis of them. In this paper we demonstrate connections between mean field theory methods and other approaches, in particular, barrier function and interior point methods. As an explicit example, we summarize our work on the linear assignment problem. In this previous work we defined a number of algorithms, including deterministic annealing, for solving the assignment problem. We proved convergence, gave bounds on the convergence times, and showed relations to other optimization algorithms.

[1]  D. R. Fulkerson,et al.  Flows in Networks. , 1964 .

[2]  Richard Sinkhorn.  A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices , 1964 .

[3]  E. Wasserstrom.  Numerical Solutions by the Continuation Method , 1973 .

[4]  L. G. H. Cijan.  A polynomial algorithm in linear programming , 1979 .

[5]  S. Ullman.  The Interpretation of Visual Motion , 1979 .

[6]  L. Khachiyan.  Polynomial algorithms in linear programming , 1980 .

[7]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[8]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Narendra Karmarkar,et al.  A new polynomial-time algorithm for linear programming , 1984, Comb..

[10]  Michael A. Saunders,et al.  On projected newton barrier methods for linear programming and an equivalence to Karmarkar’s projective method , 1986, Math. Program..

[11]  C Koch,et al.  Analog "neuronal" networks in early vision. , 1986, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Geoffrey E. Hinton,et al.  Separating Figure from Ground with a Parallel Network , 1986, Perception.

[13]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[14]  John C. Platt,et al.  Analog decoding using neural networks , 1987 .

[15]  R. Brockett,et al.  Dynamical systems that sort lists, diagonalize matrices and solve linear programming problems , 1988, Proceedings of the 27th IEEE Conference on Decision and Control.

[16]  N M Grzywacz,et al.  Massively parallel implementations of theories for apparent motion. , 1988, Spatial vision.

[17]  R. C. Monteiro,et al.  Interior path following primal-dual algorithms , 1988 .

[18]  Carsten Peterson,et al.  A New Method for Mapping Optimization Problems Onto Neural Networks , 1989, Int. J. Neural Syst..

[19]  Richard Szeliski,et al.  An Analysis of the Elastic Net Approach to the Traveling Salesman Problem , 1989, Neural Computation.

[20]  D. Bayer,et al.  The nonlinear geometry of linear programming. II. Legendre transform coordinates and central trajectories , 1989 .

[21]  D. Bayer,et al.  The Non-Linear Geometry of Linear Pro-gramming I: A?ne and projective scaling trajectories , 1989 .

[22]  Renato D. C. Monteiro,et al.  Interior path following primal-dual algorithms. part I: Linear programming , 1989, Math. Program..

[23]  Dimitri P. Bertsekas,et al.  The Auction Algorithm for Assignment and Other Network Flow Problems: A Tutorial , 1990 .

[24]  Mahesan Niranjan,et al.  A theoretical investigation into the performance of the Hopfield model , 1990, IEEE Trans. Neural Networks.

[25]  Alan L. Yuille,et al.  Generalized Deformable Models, Statistical Physics, and Matching Problems , 1990, Neural Computation.

[26]  Carsten Peterson,et al.  Parallel Distributed Approaches to Combinatorial Optimization: Benchmark Studies on Traveling Salesman Problem , 1990, Neural Computation.

[27]  Geoffrey C. Fox,et al.  A deterministic annealing approach to clustering , 1990, Pattern Recognit. Lett..

[28]  V. Braitenberg.  Reading the structure of brains , 1990 .

[29]  Federico Girosi,et al.  Parallel and deterministic algorithms from MRFs: surface reconstruction and integration , 1990, ECCV.

[30]  Petar D. Simic,et al.  Statistical mechanics as the underlying theory of ‘elastic’ and ‘neural’ optimisations , 1990 .

[31]  Federico Girosi,et al.  Parallel and Deterministic Algorithms from MRFs: Surface Reconstruction , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  L. Faybusovich,et al.  Interior-point methods and entropy , 1991, [1991] Proceedings of the 30th IEEE Conference on Decision and Control.

[33]  Margaret H. Wright,et al.  Interior methods for constrained optimization , 1992, Acta Numerica.

[34]  A. Yuille,et al.  Track finding with deformable templates — the elastic arms approach , 1992 .

[35]  Ibrahim M. Elfadel,et al.  From random fields to networks , 1993 .

[36]  Alan L. Yuille,et al.  The invisible hand algorithm: Solving the assignment problem with statistical physics , 1994, Neural Networks.

[37]  K. Urahama.  Mathematical programming formulations for neural combinatorial optimization algorithms , 1995 .

[38]  Deco,et al.  Information theory and local learning rules in a self-organizing network of Ising spins. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[39]  Ibrahim M. Elfadel,et al.  Convex Potentials and their Conjugates in Analog Mean-Field Optimization , 1995, Neural Computation.

[40]  Eric Mjolsness,et al.  Learning with Preknowledge: Clustering with Point and Graph Matching Distance Measures , 1996, Neural Computation.