Critical Temperatures for Intermittent Search in Self-Organizing Neural Networks

Kwok and Smith [1] recently proposed a new kind of optimization dynamics using self-organizing neural networks (SONN) driven by softmax weight renormalization. Such dynamics is capable of powerful intermittent search for high-quality solutions in difficult assignment optimization problems. However, the search is sensitive to temperature setting in the softmax renormalization step of the SONN algorithm. It has been hypothesized that the optimal temperature setting corresponds to symmetry breaking bifurcation of equilibria of the renormalization step, when viewed as an autonomous dynamical system called iterative softmax (ISM). We rigorously analyze equilibria of ISM by determining their number, position and stability types. Moreover, we offer analytical approximations to the critical symmetry breaking bifurcation temperatures that are in good agreement with those found by numerical investigations. So far the critical temperatures have been determined only via numerical simulations. On a set of N-queens problems for a wide range of problem sizes N, the analytically determined critical temperatures predict the optimal working temperatures for SONN intermittent search very well.