Self-annealing and self-annihilation: unifying deterministic annealing and relaxation labeling

Abstract Deterministic annealing and relaxation labeling algorithms for classification and matching are presented and discussed. A new approach – self annealing – is introduced to bring deterministic annealing and relaxation labeling into accord. Self-annealing results in an emergent linear schedule for winner-take-all and linear assignment problems. Self-annihilation, a generalization of self-annealing is capable of performing the useful function of symmetry breaking. The original relaxation labeling algorithm is then shown to arise from an approximation to either the self-annealing energy function or the corresponding dynamical system. With this relationship in place, self-annihilation can be introduced into the relaxation labeling framework. Experimental results on synthetic matching and labeling problems clearly demonstrate the three-way relationship between deterministic annealing, relaxation labeling and self-annealing.

[1]  Petar D. Simic,et al.  Statistical mechanics as the underlying theory of ‘elastic’ and ‘neural’ optimisations , 1990 .

[2]  M. Pelillo.  Relaxation labeling networks for the maximum clique problem , 1996 .

[3]  Josef Kittler,et al.  Discrete relaxation , 1990, Pattern Recognit..

[4]  Alan L. Yuille,et al.  A Convergence Proof for the Softassign Quadratic Assignment Algorithm , 1996, NIPS.

[5]  J. J. Kosowsky,et al.  Statistical Physics Algorithms That Converge , 1994, Neural Computation.

[6]  Richard Sinkhorn.  A Relationship Between Arbitrary Positive Matrices and Doubly Stochastic Matrices , 1964 .

[7]  Steven Gold,et al.  A Graduated Assignment Algorithm for Graph Matching , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Olivier D. Faugeras,et al.  Improving Consistency and Reducing Ambiguity in Stochastic Labeling: An Optimization Approach , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  L. Kitchen.  Discrete Relaxation for Matching Relational Structures , 1978 .

[10]  James S. Duncan,et al.  A robust point-matching algorithm for autoradiograph alignment , 1997, Medical Image Anal..

[11]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[12]  Alan L. Yuille,et al.  The invisible hand algorithm: Solving the assignment problem with statistical physics , 1994, Neural Networks.

[13]  Marcello Pelillo,et al.  On the dynamics of relaxation labeling processes , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[14]  Eric Mjolsness,et al.  A Lagrangian relaxation network for graph matching , 1996, IEEE Trans. Neural Networks.

[15]  Carsten Peterson,et al.  A New Method for Mapping Optimization Problems Onto Neural Networks , 1989, Int. J. Neural Syst..

[16]  Larry S. Davis,et al.  Shape Matching Using Relaxation Techniques , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Shmuel Peleg,et al.  A New Probabilistic Relaxation Scheme , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  J. G. Taylor,et al.  ARTIFICIAL NEURAL NETWORKS, 2 , 1992 .

[19]  Kiichi Urahama,et al.  Gradient Projection Network: Analog Solver for Linearly Constrained Nonlinear Programming , 1996, Neural Computation.

[20]  James S. Duncan,et al.  A Robust Point Matching Algorithm for Autoradiograph Alignment , 1996, VBC.

[21]  Eric Mjolsness,et al.  New Algorithms for 2D and 3D Point Matching: Pose Estimation and Correspondence , 1998, NIPS.

[22]  Manfred K. Warmuth,et al.  Additive versus exponentiated gradient updates for linear prediction , 1995, STOC '95.

[23]  Keith E. Price,et al.  Relaxation Matching Techniques-A Comparison , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Steven W. Zucker,et al.  On the Foundations of Relaxation Labeling Processes , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Petar D. Simic.  Constrained Nets for Graph Matching and Other Quadratic Assignment Problems , 1991, Neural Comput..

[26]  Azriel Rosenfeld,et al.  Point pattern matching by relaxation , 1980, Pattern Recognit..

[27]  W. Wolfe,et al.  Hopfield-style neural networks and the TSP , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[28]  Azriel Rosenfeld,et al.  Scene Labeling by Relaxation Operations , 1976, IEEE Transactions on Systems, Man, and Cybernetics.

[29]  Joachim M. Buhmann,et al.  Pairwise Data Clustering by Deterministic Annealing , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Eric Mjolsness,et al.  A Novel Optimizing Network Architecture with Applications , 1996, Neural Computation.

[31]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[32]  Marcello Pelillo,et al.  Learning Compatibility Coefficients for Relaxation Labeling Processes , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  John S. Bridle,et al.  Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters , 1989, NIPS.

[34]  David E. van den Bout,et al.  Graph partitioning using annealed neural networks , 1990, International 1989 Joint Conference on Neural Networks.

[35]  J. Besag.  On the Statistical Analysis of Dirty Pictures , 1986 .

[36]  Alan L. Yuille,et al.  Generalized Deformable Models, Statistical Physics, and Matching Problems , 1990, Neural Computation.

[37]  Alan L. Yuille,et al.  Statistical Physics, Mixtures of Distributions, and the EM Algorithm , 1994, Neural Computation.

[38]  G. B. Smith,et al.  Preface to S. Geman and D. Geman, “Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images” , 1987 .

[39]  K. Urahama.  Mathematical programming formulations for neural combinatorial optimization algorithms , 1995 .

[40]  Geoffrey C. Fox,et al.  Constrained Clustering as an Optimization Method , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[41]  William J. Christmas,et al.  Structural Matching in Computer Vision Using Probabilistic Relaxation , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Behrooz Kamgar-Parsi,et al.  On problem solving with Hopfield neural networks , 1990, International 1989 Joint Conference on Neural Networks.

[43]  Joachim M. Buhmann,et al.  Central and Pairwise Data Clustering by Competitive Neural Networks , 1993, NIPS.

[44]  Andrew H. Gee,et al.  Polyhedral Combinatorics and Neural Networks , 1994, Neural Computation.

[45]  Manfred K. Warmuth,et al.  Exponentiated Gradient Versus Gradient Descent for Linear Predictors , 1997, Inf. Comput..

[46]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[47]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.