Learning Global Optimization Through a Support Vector Machine Based Adaptive Multistart Strategy

We propose a global optimization algorithm called GOSAM (Global Optimization using Support vector regression based Adaptive Multistart) that applies statistical machine learning techniques, viz. Support Vector Regression (SVR) to adaptively direct iterative search in large-scale global optimization. At each iteration, GOSAM builds a training set of the objective function’s local minima discovered till the current iteration, and applies SVR to construct a regressor that learns the structure of the local minima. In the next iteration the search for the local minimum is started from the minimum of this regressor. The idea is that the regressor for local minima will generalize well to the local minima not obtained so far in the search, and hence its minimum would be a ‘crude approximation’ to the global minimum. This approximation improves over time, leading the search towards regions that yield better local minima and eventually the global minimum. Simulation results on well known benchmark problems show that GOSAM requires significantly fewer function evaluations to reach the global optimum, in comparison with methods like Particle Swarm optimization and Genetic Algorithms. GOSAM proves to be relatively more efficient as the number of design variables (dimension) increases. GOSAM does not require explicit knowledge of the objective function, and also does not assume any specific properties. We also discuss some real world applications of GOSAM involving constrained and design optimization problems.

[1]  Michael F. P. O'Boyle,et al.  Using machine learning to focus iterative optimization , 2006, International Symposium on Code Generation and Optimization (CGO'06).

[2]  Robert B. Litterman,et al.  Global Portfolio Optimization , 1992 .

[3]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[4]  Andrew W. Moore,et al.  Learning evaluation functions for global optimization , 1998 .

[5]  Aimo A. Törn,et al.  Global Optimization , 1999, Science.

[6]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[7]  Carmen G. Moles,et al.  Parameter estimation in biochemical pathways: a comparison of global optimization methods. , 2003, Genome research.

[8]  Jayadeva,et al.  Learning To Optimize VLSI Design Problems , 2006, 2006 Annual IEEE India Conference.

[9]  Olvi L. Mangasarian,et al.  Nonlinear Programming , 1969 .

[10]  Zhaoyu Wang,et al.  Global versus Local Optimization in Redundancy Resolution of Robotic Manipulators , 1988, Int. J. Robotics Res..

[11]  Marco Dorigo,et al.  Ant system: optimization by a colony of cooperating agents , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[12]  R. Horst,et al.  Global Optimization: Deterministic Approaches , 1992 .

[13]  I. Grossmann Global Optimization in Engineering Design , 2010 .

[14]  T. Kohonen,et al.  Bibliography of Self-Organizing Map SOM) Papers: 1998-2001 Addendum , 2003 .

[15]  Stefan Schaal,et al.  Natural Actor-Critic , 2003, Neurocomputing.

[16]  S. Gunn Support Vector Machines for Classification and Regression , 1998 .

[17]  J. K. Lenstra,et al.  Local Search in Combinatorial Optimisation. , 1997 .

[18]  Andrew B. Kahng,et al.  A new adaptive multi-start technique for combinatorial global optimizations , 1994, Oper. Res. Lett..

[19]  M Brady,et al.  SPECIAL ISSUE ON SENSOR DATA FUSION - FOREWORD , 1988 .

[20]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[21]  Andrew W. Moore,et al.  Learning Evaluation Functions for Global Optimization and Boolean Satisfiability , 1998, AAAI/IAAI.

[22]  Antoine Bordes,et al.  The Huller: A Simple and Efficient Online SVM , 2005, ECML.

[23]  Majid Sarrafzadeh,et al.  Congestion minimization during placement , 2000, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[24]  John C. Platt,et al.  Fast training of support vector machines using sequential minimal optimization, advances in kernel methods , 1999 .

[26]  Jayadeva,et al.  Compact analogue neural network: a new paradigm for neural based combinatorial optimisation , 1999 .

[27]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[28]  Christopher R. Houck,et al.  A Genetic Algorithm for Function Optimization: A Matlab Implementation , 2001 .