The Algorithm Selection Problem on the Continuous Optimization Domain

The problem of algorithm selection, that is identifying the most efficient algorithm for a given computational task, is non-trivial. Meta-learning techniques have been used successfully for this problem in particular domains, including pattern recognition and constraint satisfaction. However, there has been a paucity of studies focused specifically on algorithm selection for continuous optimization problems. This may be attributed to some extent to the difficulties associated with quantifying problem “hardness” in terms of the underlying cost function. In this paper, we provide a survey of the related literature in the continuous optimization domain. We discuss alternative approaches for landscape analysis, algorithm modeling and portfolio development. Finally, we propose a meta-learning framework for the algorithm selection problem in the continuous optimization domain.

[1]  Bernd Bischl,et al.  Exploratory landscape analysis , 2011, GECCO '11.

[2]  Shigeyoshi Tsutsui,et al.  Advances in evolutionary computing: theory and applications , 2003 .

[3]  Emily Anderson,et al.  Markov chain modelling of the solution surface in local search , 2002, J. Oper. Res. Soc..

[4]  Leonardo Vanneschi.  Investigating Problem Hardness of Real Life Applications , 2008 .

[5]  Peter van Emde Boas,et al.  SOFSEM 2004: Theory and Practice of Computer Science , 2004, Lecture Notes in Computer Science.

[6]  Riccardo Poli,et al.  Kolmogorov complexity, Optimization and Hardness , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[7]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[8]  Thomas Bartz-Beielstein,et al.  Analysis of Particle Swarm Optimization Using Computational Statistics , 2004 .

[9]  J. Christopher Beck,et al.  Low-Knowledge Algorithm Control , 2004, AAAI.

[10]  J. Christopher Beck,et al.  APPLYING MACHINE LEARNING TO LOW‐KNOWLEDGE CONTROL OF OPTIMIZATION ALGORITHMS , 2005, Comput. Intell..

[11]  Frédéric Benhamou.  Principles and Practice of Constraint Programming - CP 2006, 12th International Conference, CP 2006, Nantes, France, September 25-29, 2006, Proceedings , 2006, CP.

[12]  Kevin Leyton-Brown,et al.  Tradeoffs in the empirical evaluation of competing algorithm designs , 2010, Annals of Mathematics and Artificial Intelligence.

[13]  Stephen F. Smith,et al.  Combining Multiple Heuristics Online , 2007, AAAI.

[14]  E D Weinberger,et al.  Why some fitness landscapes are fractal. , 1993, Journal of theoretical biology.

[15]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[16]  Cyril Fonlupt,et al.  A Bit-Wise Epistasis Measure for Binary Search Spaces , 1998, PPSN.

[17]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[18]  Galina Merkuryeva,et al.  Structural Analysis of Benchmarking Fitness Landscapes , 2010, Sci. J. Riga Tech. Univ. Ser. Comput. Sci..

[19]  L. Darrell Whitley,et al.  Predicting Epistasis from Mathematical Models , 1999, Evolutionary Computation.

[20]  Kenneth DeJong,et al.  Parameter Setting in EAs: a 30 Year Perspective , 2007, Parameter Setting in Evolutionary Algorithms.

[21]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[22]  U. Aickelin,et al.  Parallel Problem Solving from Nature - PPSN VIII , 2004, Lecture Notes in Computer Science.

[23]  Thomas Bäck,et al.  Parallel Problem Solving from Nature — PPSN V , 1998, Lecture Notes in Computer Science.

[24]  Leonardo Vanneschi,et al.  A Study of Fitness Distance Correlation as a Difficulty Measure in Genetic Programming , 2005, Evolutionary Computation.

[25]  E. Weinberger,et al.  Correlated and uncorrelated fitness landscapes and how to tell the difference , 1990, Biological Cybernetics.

[26]  Yoav Shoham,et al.  Empirical hardness models: Methodology and a case study on combinatorial auctions , 2009, JACM.

[27]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[28]  Ryan Williams,et al.  Confronting hardness using a hybrid approach , 2006, SODA '06.

[29]  Julian Francis Miller,et al.  Information Characteristics and the Structure of Landscapes , 2000, Evolutionary Computation.

[30]  Sébastien Vérel,et al.  Fitness Clouds and Problem Hardness in Genetic Programming , 2004, GECCO.

[31]  Thomas Bartz-Beielstein,et al.  Tuning search algorithms for real-world applications: a regression tree based approach , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[32]  Mario A. Muñoz,et al.  A Meta-learning Prediction Model of Algorithm Performance for Continuous Optimization Problems , 2012, PPSN.

[33]  Olivier François,et al.  Design of evolutionary algorithms-A statistical perspective , 2001, IEEE Trans. Evol. Comput..

[34]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms: An Initial Investigation ∗ , 2006 .

[35]  Assaf Naor,et al.  Rigorous location of phase transitions in hard optimization problems , 2005, Nature.

[36]  L. Darrell Whitley,et al.  Problem difficulty for tabu search in job-shop scheduling , 2003, Artif. Intell..

[37]  Riccardo Poli,et al.  Limitations of the fitness-proportional negative slope coefficient as a difficulty measure , 2009, GECCO '09.

[38]  Susan L. Epstein,et al.  Learning a Mixture of Search Heuristics , 2012, Autonomous Search.

[39]  Edmund H. Durfee,et al.  Using Landscape Theory to Measure Learning Difficulty for Adaptive Agents , 2002, Adaptive Agents and Multi-Agents Systems.

[40]  Daniel Kudenko,et al.  Adaptive Agents and Multi-Agent Systems , 2003, Lecture Notes in Computer Science.

[41]  Thomas Jansen,et al.  Design and Management of Complex Technical Processes and Systems by Means of Computational Intelligence Methods on Classifications of Fitness Functions on Classifications of Fitness Functions , 2022 .

[42]  Colin R. Reeves,et al.  An Experimental Design Perspective on Genetic Algorithms , 1994, FOGA.

[43]  Jano I. van Hemert,et al.  Discovering the suitability of optimisation algorithms by learning from evolved instances , 2011, Annals of Mathematics and Artificial Intelligence.

[44]  Phil Husbands,et al.  Fitness Landscapes and Evolvability , 2002, Evolutionary Computation.

[45]  M. Victoria Luzón,et al.  Modeling the Performance of Evolutionary Algorithms on the Root Identification Problem: A Case Study with PBIL and CHC Algorithms , 2011, Evolutionary Computation.

[46]  M. E. H. Pedersen,et al.  Tuning & simplifying heuristical optimization , 2010 .

[47]  Bart Naudts,et al.  Epistasis as a Basic Concept in Formal Landscape Analysis , 1997, ICGA.

[48]  Riccardo Poli,et al.  Information landscapes , 2005, GECCO '05.

[49]  Riccardo Poli,et al.  Fitness Distributions and GA Hardness , 2004, PPSN.

[50]  Anton V. Eremeev,et al.  Non-parametric Estimation of Properties of Combinatorial Landscapes , 2002, EvoWorkshops.

[51]  Riccardo Poli,et al.  Genetic and Evolutionary Computation – GECCO 2004 , 2004, Lecture Notes in Computer Science.

[52]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms , 2006, CP.

[53]  L. Darrell Whitley,et al.  The dispersion metric and the CMA evolution strategy , 2006, GECCO.

[54]  Nicolas Barnier,et al.  Solving the Kirkman's schoolgirl problem in a few seconds , 2002 .

[55]  Vesselin K. Vassilev,et al.  Smoothness, ruggedness and neutrality of fitness landscapes: from theory to application , 2003 .

[56]  Mohamed Slimane,et al.  A Critical and Empirical Study of Epistasis Measures for Predicting GA Performances: A Summary , 1997, Artificial Evolution.

[57]  Ender Özcan,et al.  Hill Climbers and Mutational Heuristics in Hyperheuristics , 2006, PPSN.

[58]  Bruce A. Robinson,et al.  Self-Adaptive Multimethod Search for Global Optimization in Real-Parameter Spaces , 2009, IEEE Transactions on Evolutionary Computation.

[59]  Fei Peng,et al.  Population-Based Algorithm Portfolios for Numerical Optimization , 2010, IEEE Transactions on Evolutionary Computation.

[60]  Terry Jones,et al.  Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms , 1995, ICGA.

[61]  Terence Soule,et al.  Genetic Programming Theory and Practice XVI , 2015, Genetic and Evolutionary Computation.

[62]  Patrick De Causmaecker,et al.  Towards prediction of algorithm performance in real world optimisation problems , 2009 .

[63]  Dong-il Seo,et al.  An Information-Theoretic Analysis on the Interactions of Variables in Combinatorial Optimization Problems , 2007, Evolutionary Computation.

[64]  Christian L. Müller,et al.  Global Characterization of the CEC 2005 Fitness Landscapes Using Fitness-Distance Analysis , 2011, EvoApplications.

[65]  Constantin Halatsis,et al.  Measures of Intrinsic Hardness for Constraint Satisfaction Problem Instances , 2004, SOFSEM.

[66]  Lloyd Dudley Fosdick.  Performance evaluation of numerical software : proceedings of the IFIP TC 2.5 Working Conference on Performance Evaluation of Numerical Software , 1979 .

[67]  Marc Schoenauer,et al.  Artificial Evolution , 2000, Lecture Notes in Computer Science.

[68]  Riccardo Poli,et al.  Practical performance models of algorithms in evolutionary program induction and other domains , 2010, Artif. Intell..

[69]  Yuval Davidor,et al.  Epistasis Variance: A Viewpoint on GA-Hardness , 1990, FOGA.

[70]  Zbigniew Michalewicz,et al.  Parameter control in evolutionary algorithms , 1999, IEEE Trans. Evol. Comput..

[71]  Anton V. Eremeev,et al.  On Confidence Intervals for the Number of Local Optima , 2003, EvoWorkshops.

[72]  Josselin Garnier,et al.  Efficiency of Local Search with Multiple Local Optima , 2001, SIAM J. Discret. Math..

[73]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[74]  Sébastien Vérel,et al.  Negative Slope Coefficient: A Measure to Characterize Genetic Programming Fitness Landscapes , 2006, EuroGP.

[75]  Peter F. Stadler,et al.  Fast Fourier Transform for Fitness Landscapes , 2002 .

[76]  Vassilis Zissimopoulos,et al.  On the Hardness of the Quadratic Assignment Problem with Metaheuristics , 2002, J. Heuristics.

[77]  J. Christopher Beck,et al.  Adaptive Search Algorithms and Fitness-Distance Correlation , 2003 .

[78]  Patricia Diane Hough,et al.  Modern Machine Learning for Automatic Optimization Algorithm Selection. , 2006 .

[79]  Mario A. Muñoz,et al.  Landscape characterization of numerical optimization problems using biased scattered data , 2012, 2012 IEEE Congress on Evolutionary Computation.

[80]  Kousha Etessami,et al.  Recursive Markov chains, stochastic grammars, and monotone systems of nonlinear equations , 2005, JACM.

[81]  Xin Yao,et al.  A Note on Problem Difficulty Measures in Black-Box Optimization: Classification, Realizations and Predictability , 2007, Evolutionary Computation.

[82]  Hussein A. Abbass,et al.  Motif Difficulty (MD): A Predictive Measure of Problem Difficulty for Evolutionary Algorithms Using Network Motifs , 2012, Evolutionary Computation.

[83]  Edmund K. Burke,et al.  Parallel Problem Solving from Nature - PPSN IX: 9th International Conference, Reykjavik, Iceland, September 9-13, 2006, Proceedings , 2006, PPSN.

[84]  Rolf Drechsler,et al.  Applications of Evolutionary Computing, EvoWorkshops 2008: EvoCOMNET, EvoFIN, EvoHOT, EvoIASP, EvoMUSART, EvoNUM, EvoSTOC, and EvoTransLog, Naples, Italy, March 26-28, 2008. Proceedings , 2008, EvoWorkshops.

[85]  Bart Naudts,et al.  A comparison of predictive measures of problem difficulty in evolutionary algorithms , 2000, IEEE Trans. Evol. Comput..

[86]  Anton V. Eremeev,et al.  Statistical analysis of local search landscapes , 2004, J. Oper. Res. Soc..

[87]  Andries Petrus Engelbrecht,et al.  Alternative hyper-heuristic strategies for multi-method global optimization , 2010, IEEE Congress on Evolutionary Computation.

[88]  Francisco Herrera,et al.  Adaptive local search parameters for real-coded memetic algorithms , 2005, 2005 IEEE Congress on Evolutionary Computation.

[89]  P. Stadler.  Landscapes and their correlation functions , 1996 .

[90]  Jürgen Branke,et al.  Faster convergence by means of fitness estimation , 2005, Soft Comput..

[91]  Anthony Brabazon,et al.  Defining locality as a problem difficulty measure in genetic programming , 2011, Genetic Programming and Evolvable Machines.