Adaptive global optimization with local search

This dissertation examines the performance of genetic algorithm (GA) hybrids that use local search to solve global optimization problems. GAs are a class of adaptive global sampling methods that take many cues from mechanisms observed in natural evolution. GAs maintain a population of solutions that are used to generate new solutions in the search space. GA hybrids using local search (GA-LS hybrids) are motivated by the apparent need to employ both a global and local search strategy to provide an effective global optimization method. Previous experimental results have found that GA-LS hybrids not only find better solutions than the GA, but also optimize more efficiently. To improve the efficiency of GA-LS hybrids, I propose and experimentally validate methods that selectively apply local search to solutions in the GA's population. First, local search is randomly applied with a fixed frequency. Experiments with this method illustrate a trade-off between the refinement performed by local search and the reliability of the competitive search performed by the GA. Next, I describe two classes of adaptive methods. Distribution-based adaptive methods use redundancy in the population to avoid performing unnecessary local searches. Fitness-based adaptive methods use the fitness information in the population to bias the local search towards individuals that have better fitness. An experimental analysis indicates that these adaptive methods can significantly improve the efficiency of GA-LS hybrids. This dissertation explores implications of these results for parallel GAs. In particular, a MIMD design for geographically structured genetic algorithms (GSGAs) is described. GSGAs were initially developed for SIMD architectures, where it is difficult to selectively apply local search. An analytic and experimental analysis of MIMD GSGAs demonstrates that they scale well for large problems. GA-LS hybrids are used to solve global optimization problems in several application domains. First, GA-LS hybrids are used to find the weights of a neural network to solve the six-bit symmetry problem. Next, they are used to solve a simple 19 atom molecular conformation problem. Finally, they are applied to a drug docking problem. When compared to simulated annealing, GA-LS hybrids not only find better solutions and optimize more efficiently.

[1]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[2]  John T. Gill,et al.  Computational complexity of probabilistic Turing machines , 1974, STOC '74.

[3]  Peter E. Hart,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[4]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[5]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[6]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[7]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[8]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[9]  Roger J.-B. Wets,et al.  Minimization by Random Search Techniques , 1981, Math. Oper. Res..

[10]  Philip E. Gill,et al.  Practical optimization , 1981 .

[11]  D. Hartl,et al.  A primer of population genetics , 1981 .

[12]  H. Niederreiter,et al.  A quasi-Monte Carlo method for the approximate computation of the extreme values of a function , 1983 .

[13]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[14]  Harald Niederreiter,et al.  Quasi-Monte Carlo Methods for Global Optimization , 1985 .

[15]  Mihalis Yannakakis,et al.  How easy is local search? , 1985, 26th Annual Symposium on Foundations of Computer Science (sfcs 1985).

[16]  P. Goodford.  A computational procedure for determining energetically favorable binding sites on biologically important macromolecules. , 1985, Journal of medicinal chemistry.

[17]  Alan Weiss,et al.  Allocating Independent Subtasks on Parallel Processors , 1985, IEEE Transactions on Software Engineering.

[18]  C. Tovey.  Hill Climbing with Multiple Local Optima , 1985 .

[19]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[20]  H. Niederreiter,et al.  Localization of Search in Quasi-Monte Carlo Methods for Global Optimization , 1986 .

[21]  Bibliography of theoretical calculations in molecular pharmacology , 1986 .

[22]  Geoffrey E. Hinton,et al.  A general framework for parallel distributed processing , 1986 .

[23]  C. Dorea,et al.  Limiting distribution for random optimization methods , 1986 .

[24]  Craig A. Tovey,et al.  Low order polynomial bounds on the expected performance of local improvement algorithms , 1986, Math. Program..

[25]  G. T. Timmer,et al.  Stochastic global optimization methods part I: Clustering methods , 1987, Math. Program..

[26]  Alexander H. G. Rinnooy Kan,et al.  Bayesian stopping rules for multistart global optimization methods , 1987, Math. Program..

[27]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[28]  Henryk Wozniakowski,et al.  Information-based complexity , 1987, Nature.

[29]  G. T. Timmer,et al.  Stochastic global optimization methods part II: Multi level methods , 1987, Math. Program..

[30]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[31]  D. Ackley.  A connectionist machine for genetic hillclimbing , 1987 .

[32]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[33]  Heinz Mühlenbein,et al.  Evolution algorithms in combinatorial optimization , 1988, Parallel Comput..

[34]  N. E. Collins,et al.  Simulated annealing - an annotated bibliography , 1988 .

[35]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[36]  Rajarshi Das,et al.  A Study of Control Parameters Affecting Online Performance of Genetic Algorithms for Function Optimization , 1989, ICGA.

[37]  R. Cramer,et al.  Validation of the general purpose tripos 5.2 force field , 1989 .

[38]  Gilbert Syswerda,et al.  Uniform Crossover in Genetic Algorithms , 1989, ICGA.

[39]  Halbert White,et al.  Learning in Artificial Neural Networks: A Statistical Perspective , 1989, Neural Computation.

[40]  L. Ingber.  Very fast simulated re-annealing , 1989 .

[41]  David E. Goldberg,et al.  Genetic Algorithms and Walsh Functions: Part I, A Gentle Introduction , 1989, Complex Syst..

[42]  Mihalis Yannakakis,et al.  On the complexity of local search , 1990, STOC '90.

[43]  C. Dorea.  Stopping rules for a random optimization method , 1990 .

[44]  D. Goodsell,et al.  Automated docking of substrates to proteins by simulated annealing , 1990, Proteins.

[45]  Kenneth A. De Jong,et al.  An Analysis of Multi-Point Crossover , 1990, FOGA.

[46]  David G. Stork,et al.  Evolution and Learning in Neural Networks , 1990, NIPS.

[47]  David E. Goldberg,et al.  The Theory of Virtual Alphabets , 1990, PPSN.

[48]  Panos M. Pardalos,et al.  A Collection of Test Problems for Constrained Global Optimization Algorithms , 1990, Lecture Notes in Computer Science.

[49]  Peter Neuhaus,et al.  Solving the Mapping Problem - Experiences with a Genetic Algorithm , 1990, PPSN.

[50]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[51]  Heinrich Braun,et al.  On Solving Travelling Salesman Problems by Genetic Algorithms , 1990, PPSN.

[52]  Richard K. Belew,et al.  Evolution, Learning, and Culture: Computational Metaphors for Adaptive Algorithms , 1990, Complex Syst..

[53]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[54]  Emile H. L. Aarts,et al.  Genetic Local Search Algorithms for the Travelling Salesman Problem , 1990, PPSN.

[55]  David S. Johnson,et al.  Local Optimization and the Traveling Salesman Problem , 1990, ICALP.

[56]  Alden H. Wright,et al.  Genetic Algorithms for Real Parameter Optimization , 1990, FOGA.

[57]  Kalyanmoy Deb,et al.  A Comparative Analysis of Selection Schemes Used in Genetic Algorithms , 1990, FOGA.

[58]  Heinz Mühlenbein,et al.  The parallel genetic algorithm as function optimizer , 1991, Parallel Comput..

[59]  L. Darrell Whitley,et al.  Genetic Reinforcement Learning with Multilayer Neural Networks , 1991, ICGA.

[60]  L. Darrell Whitley,et al.  Delta Coding: An Iterative Search Strategy for Genetic Algorithms , 1991, ICGA.

[61]  Thomas Bäck,et al.  Extended Selection Mechanisms in Genetic Algorithms , 1991, ICGA.

[62]  L. Toothaker.  Multiple Comparisons for Researchers , 1991 .

[63]  Thomas Bäck,et al.  A Survey of Evolution Strategies , 1991, ICGA.

[64]  Zbigniew Michalewicz,et al.  An Experimental Comparison of Binary and Floating Point Representations in Genetic Algorithms , 1991, ICGA.

[65]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[66]  David R. Jefferson,et al.  Selection in Massively Parallel Genetic Algorithms , 1991, ICGA.

[67]  Schloss Birlinghoven.  Evolution in Time and Space -the Parallel Genetic Algorithm , 1991 .

[68]  Bernard Manderick,et al.  A Massively Parallel Genetic Algorithm: Implementation and First Analysis , 1991, ICGA.

[69]  J. S. Cramer,et al.  The Logit Model: An Introduction for Economists , 1991 .

[70]  William E. Hart,et al.  Optimizing an Arbitrary Function is Hard for the Genetic Algorithm , 1991 .

[71]  Gunar E. Liepins,et al.  Schema Disruption , 1991, ICGA.

[72]  Melanie Mitchell,et al.  The Performance of Genetic Algorithms on Walsh Polynomials: Some Anomalous Results and Their Explanation , 1991, ICGA.

[73]  Melanie Mitchell,et al.  The royal road for genetic algorithms: Fitness landscapes and GA performance , 1991 .

[74]  Gregor von Laszewski,et al.  Intelligent Structural Operators for the k-way Graph Partitioning Problem , 1991, ICGA.

[75]  Bruce E. Rosen,et al.  Genetic Algorithms and Very Fast Simulated Reannealing: A comparison , 1992 .

[76]  Juan C. Meza,et al.  Do intelligent configuration search techniques outperform random search for large molecules , 1992 .

[77]  John Michael Neal McInerney,et al.  Biologically influenced algorithms and parallelism in non-linear optimization , 1992 .

[78]  J. Ben Rosen,et al.  A discrete-continuous algorithm for molecular energy minimization , 1992, Proceedings Supercomputing '92.

[79]  Takeshi Yamada,et al.  The ECOlogical Framework II : Improving GA Performance At Virtually Zero Cost , 1993, ICGA.

[80]  Kenneth M. Merz,et al.  The application of the genetic algorithm to the minimization of potential energy functions , 1993, J. Glob. Optim..

[81]  L. Darrell Whitley,et al.  Cellular Genetic Algorithms , 1993, ICGA.

[82]  L. Darrell Whitley,et al.  Serial and Parallel Genetic Algorithms as Function Optimizers , 1993, ICGA.

[83]  David B. Fogel,et al.  An introduction to simulated evolutionary optimization , 1994, IEEE Trans. Neural Networks.

[84]  Günter Rudolph,et al.  Convergence analysis of canonical genetic algorithms , 1994, IEEE Trans. Neural Networks.

[85]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[86]  Scott B. Baden,et al.  A robust parallel programming model for dynamic non-uniform scientific computations , 1994, Proceedings of IEEE Scalable High Performance Computing Conference.

[87]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[88]  William E. Hart,et al.  Optimization with genetic algorithm hybrids that use local searches , 1996 .