Equilibria of Iterative Softmax and Critical Temperatures for Intermittent Search in Self-Organizing Neural Networks

Kwok and Smith (2005) recently proposed a new kind of optimization dynamics using self-organizing neural networks (SONN) driven by softmax weight renormalization. Such dynamics is capable of powerful intermittent search for high-quality solutions in difficult assignment optimization problems. However, the search is sensitive to temperature setting in the softmax renormalization step. It has been hypothesized that the optimal temperature setting corresponds to the symmetry-breaking bifurcation of equilibria of the renormalization step, when viewed as an autonomous dynamical system called iterative softmax (ISM). We rigorously analyze equilibria of ISM by determining their number, position, and stability types. It is shown that most fixed points exist in the neighborhood of the maximum entropy equilibrium = (N1, N1, , N1), where N is the ISM dimensionality. We calculate the exact rate of decrease in the number of ISM equilibria as one moves away from . Bounds on temperatures guaranteeing different stability types of ISM equilibria are also derived. Moreover, we offer analytical approximations to the critical symmetry-breaking bifurcation temperatures that are in good agreement with those found by numerical investigations. So far, the critical temperatures have been determined only by trial-and-error numerical simulations. On a set of N-queens problems for a wide range of problem sizes N, the analytically determined critical temperatures predict the optimal working temperatures for SONN intermittent search very well. It is also shown that no intermittent search can exist in SONN for temperatures greater than one-half.

[1]  Kate Smith-Miles,et al.  Neural Networks for Combinatorial Optimization: A Review of More Than a Decade of Research , 1999, INFORMS J. Comput..

[2]  Marimuthu Palaniswami,et al.  Neural techniques for combinatorial optimization with applications , 1998, IEEE Trans. Neural Networks.

[3]  T. Kohonen.  Self-Organized Formation of Correct Feature Maps , 1982 .

[4]  Steven Minton,et al.  Minimizing Conflicts: A Heuristic Repair Method for Constraint Satisfaction and Scheduling Problems , 1992, Artif. Intell..

[5]  Kate A. Smith.  Solving the generalised quadratic assignment problem using a self-organising process , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[6]  T. Kohonen.  Self-organized formation of topology correct feature maps , 1982 .

[7]  Edward Tsang,et al.  Solving constraint satisfaction problems using neural networks , 1991 .

[8]  Kate Smith-Miles,et al.  A noisy self-organizing neural network with bifurcation dynamics for combinatorial optimization , 2004, IEEE Transactions on Neural Networks.

[9]  D. Canca,et al.  Manufacturing cell formation using a new self-organizing neural network , 2002 .

[10]  Christopher G. Langton,et al.  Computation at the edge of chaos: Phase transitions and emergent computation , 1990 .

[11]  Alan L. Yuille,et al.  The invisible hand algorithm: Solving the assignment problem with statistical physics , 1994, Neural Networks.

[12]  Steven Gold,et al.  Softmax to Softassign: neural network algorithms for combinatorial optimization , 1996 .

[13]  Kate Smith-Miles,et al.  Optimization via Intermittency with a Self-Organizing Neural Network , 2005, Neural Computation.

[14]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[15]  Anand Rangarajan,et al.  Self-annealing and self-annihilation: unifying deterministic annealing and relaxation labeling , 2000, Pattern Recognit..

[16]  James P. Crutchfield,et al.  Computation at the Onset of Chaos , 1991 .

[17]  Kate A. Smith,et al.  Characteristic updating-normalisation dynamics of a self-organising neural network for enhanced combinatorial optimisation , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[18]  John L. Wyatt,et al.  The Softmax Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element , 1993, NIPS.

[19]  Kate Smith-Miles,et al.  Performance-Enhancing Bifurcations in a Self-Organising Neural Network , 2003, IWANN.