Simulated Annealing: Rigorous finite-time guarantees for optimization on continuous domains

Simulated annealing is a popular method for approaching the solution of a global optimization problem. Existing results on its performance apply to discrete combinatorial optimization where the optimization variables can assume only a finite set of possible values. We introduce a new general formulation of simulated annealing which allows one to guarantee finite-time performance in the optimization of functions of continuous variables. The results hold universally for any optimization problem on a bounded domain and establish a connection between simulated annealing and up-to-date theory of convergence of Markov chain Monte Carlo methods on continuous domains. This work is inspired by the concept of finite-time learning with known accuracy and confidence developed in statistical learning theory.

[1]  Assaf Naor,et al.  Rigorous location of phase transitions in hard optimization problems , 2005, Nature.

[2]  E. Bonomi,et al.  The N-City Travelling Salesman Problem: Statistical Mechanics and the Metropolis Algorithm , 1984 .

[3]  D. Mitra,et al.  Convergence and finite-time behavior of simulated annealing , 1985, 1985 24th IEEE Conference on Decision and Control.

[4]  Mathukumalli Vidyasagar,et al.  Learning and Generalization: With Applications to Neural Networks , 2002 .

[5]  M. A. Best.  Bayesian Approaches to Clinical Trials and Health‐Care Evaluation , 2005 .

[6]  Sanjeev R. Kulkarni,et al.  Relative Frequencies of Generalized Simulated Annealing , 2006, Math. Oper. Res..

[7]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[8]  H. Haario,et al.  Simulated annealing process in general state space , 1991, Advances in Applied Probability.

[9]  Mathukumalli Vidyasagar,et al.  Randomized algorithms for robust controller synthesis using statistical learning theory , 2001, Autom..

[10]  Lenore Blum,et al.  Complexity and Real Computation , 1997, Springer New York.

[11]  M. Mézard,et al.  Analytic and Algorithmic Solution of Random Satisfiability Problems , 2002, Science.

[12]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[13]  Boris Gnedenko,et al.  Theory of Probability , 1963 .

[14]  Emile H. L. Aarts,et al.  Simulated Annealing: Theory and Applications , 1987, Mathematics and Its Applications.

[15]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[16]  P. Anderson,et al.  Application of statistical mechanics to NP-complete problems in combinatorial optimisation , 1986 .

[17]  M. Locatelli.  Simulated Annealing Algorithms for Continuous Global Optimization: Convergence Conditions , 2000 .

[18]  Saul B. Gelfand,et al.  Simulated annealing type algorithms for multivariate optimization , 1991, Algorithmica.

[19]  P. Müller,et al.  Optimal Bayesian Design by Inhomogeneous Markov Chain Simulation , 2004 .

[20]  R. Tempo,et al.  Randomized Algorithms for Analysis and Control of Uncertain Systems , 2004 .

[21]  J. Bernardo,et al.  Simulation-Based Optimal Design , 1999 .

[22]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization , 1997 .

[23]  Bruce E. Hajek,et al.  Cooling Schedules for Optimal Annealing , 1988, Math. Oper. Res..

[24]  C. Tsallis,et al.  Generalized simulated annealing , 1995, cond-mat/9501047.

[25]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[26]  Richard L. Tweedie,et al.  Markov Chains and Stochastic Stability , 1993, Communications and Control Engineering Series.

[27]  R. Tweedie,et al.  Rates of convergence of the Hastings and Metropolis algorithms , 1996 .

[28]  J. Rosenthal.  Minorization Conditions and Convergence Rates for Markov Chain Monte Carlo , 1995 .

[29]  J. Rosenthal,et al.  General state space Markov chains and MCMC algorithms , 2004, math/0404033.

[30]  John Lygeros,et al.  Monte Carlo Optimization for Conflict Resolution in Air Traffic Control , 2006, IEEE Transactions on Intelligent Transportation Systems.

[31]  Christian P. Robert,et al.  Monte Carlo Statistical Methods (Springer Texts in Statistics) , 2005 .

[32]  D. Mitra,et al.  Convergence and finite-time behavior of simulated annealing , 1986, Advances in Applied Probability.