Automated Algorithm Selection: Survey and Perspectives

It has long been observed that for practically any computational problem that has been intensely studied, different instances are best solved using different algorithms. This is particularly pronounced for computationally hard problems, where in most cases, no single algorithm defines the state of the art; instead, there is a set of algorithms with complementary strengths. This performance complementarity can be exploited in various ways, one of which is based on the idea of selecting, from a set of given algorithms, for each problem instance to be solved the one expected to perform best. The task of automatically selecting an algorithm from a given set is known as the per-instance algorithm selection problem and has been intensely studied over the past 15 years, leading to major improvements in the state of the art in solving a growing number of discrete combinatorial problems, including propositional satisfiability and AI planning. Per-instance algorithm selection also shows much promise for boosting performance in solving continuous and mixed discrete/continuous optimisation problems. This survey provides an overview of research in automated algorithm selection, ranging from early and seminal works to recent and promising application areas. Different from earlier work, it covers applications to discrete and continuous problems, and discusses algorithm selection in context with conceptually related approaches, such as algorithm configuration, scheduling, or portfolio selection. Since informative and cheaply computable problem instance features provide the basis for effective per-instance algorithm selection systems, we also provide an overview of such features for discrete and continuous problems. Finally, we provide perspectives on future work in the area and discuss a number of open research challenges.

[1]  Adele E. Howe,et al.  Exploiting Competitive Planner Performance , 1999, ECP.

[2]  Marius Lindauer,et al.  An Empirical Study of Per-instance Algorithm Scheduling , 2016, LION.

[3]  Hans-Peter Kriegel,et al.  Density-Based Clustering in Spatial Databases: The Algorithm GDBSCAN and Its Applications , 1998, Data Mining and Knowledge Discovery.

[4]  Saman K. Halgamuge,et al.  Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content , 2015, IEEE Transactions on Evolutionary Computation.

[5]  Yoav Shoham,et al.  Understanding Random SAT: Beyond the Clauses-to-Variables Ratio , 2004, CP.

[6]  Carlos M. Fonseca,et al.  Exploring the Performance of Stochastic Multiobjective Optimisers with the Second-Order Attainment Function , 2005, EMO.

[7]  Chitta Baral,et al.  Knowledge Representation, Reasoning and Declarative Problem Solving , 2003 .

[8]  Heike Trautmann,et al.  Evolving Instances for Maximizing Performance Differences of State-of-the-Art Inexact TSP Solvers , 2016, LION.

[9]  Torsten Schaub,et al.  AutoFolio: An Automatically Configured Algorithm Selector (Extended Abstract) , 2017, IJCAI.

[10]  Josef Pihera,et al.  Application of Machine Learning to Algorithm Selection for TSP , 2014, 2014 IEEE 26th International Conference on Tools with Artificial Intelligence.

[11]  Fernando Fernández,et al.  Learning Predictive Models to Configure Planning Portfolios , 2013 .

[12]  Frank Neumann,et al.  Feature-based algorithm selection for constrained continuous optimisation , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[13]  Heike Trautmann,et al.  Understanding Characteristics of Evolved Instances for State-of-the-Art Inexact TSP Solvers with Maximum Performance Difference , 2016, AI*IA.

[14]  D. Wolpert,et al.  No Free Lunch Theorems for Search , 1995 .

[15]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[16]  Marius Thomas Lindauer,et al.  AutoFolio: An Automatically Configured Algorithm Selector , 2015, J. Artif. Intell. Res..

[17]  Gary B. Lamont,et al.  Evolutionary Algorithms for Solving Multi-Objective Problems , 2002, Genetic Algorithms and Evolutionary Computation.

[18]  Petr Posík,et al.  Global Line Search Algorithm Hybridized with Quadratic Interpolation and Its Extension to Separable Functions , 2015, GECCO.

[19]  Heike Trautmann,et al.  Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning , 2017, Evolutionary Computation.

[20]  Lukás Chrpa,et al.  The 2014 International Planning Competition: Progress and Trends , 2015, AI Mag..

[21]  Ying Wah Teh,et al.  On Density-Based Data Streams Clustering Algorithms: A Survey , 2014, Journal of Computer Science and Technology.

[22]  Shiu Yin Yuen,et al.  On composing an algorithm portfolio , 2015, Memetic Computing.

[23]  P. Stadler Fitness Landscapes , 1993 .

[24]  Jakob Bossek Network Generator for Combinatorial Graph Problems , 2016 .

[25]  Heike Trautmann,et al.  Sliding to the global optimum: How to benefit from non-global optima in multimodal multi-objective optimization , 2019 .

[26]  Marc Schoenauer,et al.  Per instance algorithm configuration of CMA-ES with limited budget , 2017, GECCO.

[27]  Markus Wagner,et al.  A case study of algorithm selection for the traveling thief problem , 2016, Journal of Heuristics.

[28]  Thomas Stützle,et al.  On the Empirical Scaling Behaviour of State-of-the-art Local Search Algorithms for the Euclidean TSP , 2015, GECCO.

[29]  Y. Shoham,et al.  SATzilla : An Algorithm Portfolio for SAT ∗ , 2004 .

[30]  Asma Atamna,et al.  Benchmarking IPOP-CMA-ES-TPA and IPOP-CMA-ES-MSR on the BBOB Noiseless Testbed , 2015, GECCO.

[31]  Malte Helmert,et al.  The Fast Downward Planning System , 2006, J. Artif. Intell. Res..

[32]  Geoff Holmes,et al.  Algorithm Selection on Data Streams , 2014, Discovery Science.

[33]  Brian W. Kernighan,et al.  An Effective Heuristic Algorithm for the Traveling-Salesman Problem , 1973, Oper. Res..

[34]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms , 2006, CP.

[35]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[36]  Christian L. Müller,et al.  Global Characterization of the CEC 2005 Fitness Landscapes Using Fitness-Distance Analysis , 2011, EvoApplications.

[37]  Jörg Hoffmann Analyzing Search Topology Without Running Any Search: On the Connection Between Causal Graphs and h+ , 2011, J. Artif. Intell. Res..

[38]  Julian Francis Miller,et al.  Information Characteristics and the Structure of Landscapes , 2000, Evolutionary Computation.

[39]  Kevin Leyton-Brown,et al.  Evaluating Component Solver Contributions to Portfolio-Based Algorithm Selectors , 2012, SAT.

[40]  Heike Trautmann,et al.  Improving the State of the Art in Inexact TSP Solving Using Per-Instance Algorithm Selection , 2015, LION.

[41]  L. Darrell Whitley,et al.  The dispersion metric and the CMA evolution strategy , 2006, GECCO.

[42]  Bernd Bischl,et al.  mlr: Machine Learning in R , 2016, J. Mach. Learn. Res..

[43]  Bernd Bischl,et al.  A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem , 2012, Annals of Mathematics and Artificial Intelligence.

[44]  Jiming Liu,et al.  Multiagent Optimization System for Solving the Traveling Salesman Problem (TSP) , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[45]  Marcus Gallagher,et al.  Analysing and characterising optimization problems using length scale , 2017, Soft Comput..

[46]  Vassilis Zissimopoulos,et al.  On the Hardness of the Quadratic Assignment Problem with Metaheuristics , 2002, J. Heuristics.

[47]  Frank Neumann,et al.  A Feature-Based Analysis on the Impact of Set of Constraints for e-Constrained Differential Evolution , 2015, ArXiv.

[48]  T. Marius Lindauer,et al.  Algorithm Selection, Scheduling and Con guration of Boolean Constraint Solvers , 2014 .

[49]  L. Darrell Whitley,et al.  Efficient Recombination in the Lin-Kernighan-Helsgaun Traveling Salesman Heuristic , 2018, PPSN.

[50]  Terry Jones,et al.  Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms , 1995, ICGA.

[51]  Marc Schoenauer,et al.  Feature Based Algorithm Configuration: A Case Study with Differential Evolution , 2016, PPSN.

[52]  Michael T. M. Emmerich,et al.  Test Problems Based on Lamé Superspheres , 2007, EMO.

[53]  László Pál,et al.  Comparison of multistart global optimization algorithms on the BBOB noiseless testbed , 2013, GECCO.

[54]  Tea Tusar,et al.  Visualization of Pareto Front Approximations in Evolutionary Multiobjective Optimization: A Critical Review and the Prosection Method , 2015, IEEE Transactions on Evolutionary Computation.

[55]  M. Preuss,et al.  Search Dynamics on Multimodal Multiobjective Problems , 2019, Evolutionary Computation.

[56]  Lars Kotthoff,et al.  Algorithm Selection for Combinatorial Search Problems: A Survey , 2012, AI Mag..

[57]  Stuart A. Kauffman,et al.  The origins of order , 1993 .

[58]  Thomas Stützle,et al.  Exploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization , 2010, Experimental Methods for the Analysis of Optimization Algorithms.

[59]  Simon Wessing Two-stage methods for multimodal optimization , 2015 .

[60]  Hao Wang,et al.  Algorithm configuration data mining for CMA evolution strategies , 2017, GECCO.

[61]  Greg Hamerly,et al.  Learning the k in k-means , 2003, NIPS.

[62]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[63]  Yuri Malitsky,et al.  Features for Exploiting Black-Box Optimization Problem Structure , 2013, LION.

[64]  Jan Peters,et al.  Stability of Controllers for Gaussian Process Dynamics , 2017, J. Mach. Learn. Res..

[65]  T.,et al.  An experimental study of adaptive capping in irace , 2017 .

[66]  Pascal Kerschke,et al.  Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-Package Flacco , 2017, Studies in Classification, Data Analysis, and Knowledge Organization.

[67]  Edmund H. Durfee,et al.  Using Landscape Theory to Measure Learning Difficulty for Adaptive Agents , 2002, Adaptive Agents and Multi-Agents Systems.

[68]  Xin Yao,et al.  Population-based Algorithm Portfolios with automated constituent algorithms selection , 2014, Inf. Sci..

[69]  Heike Trautmann,et al.  Leveraging TSP Solver Complementarity through Machine Learning , 2018, Evolutionary Computation.

[70]  Stefan Szeider,et al.  Portfolio-Based Algorithm Selection for Circuit QBFs , 2018, CP.

[71]  Kevin Leyton-Brown,et al.  Hierarchical Hardness Models for SAT , 2007, CP.

[72]  Kate Smith-Miles,et al.  Towards insightful algorithm selection for optimisation using meta-learning concepts , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[73]  Thomas Stützle,et al.  AClib: A Benchmark Library for Algorithm Configuration , 2014, LION.

[74]  Fernando Fernández,et al.  IBACOP and IBACOP2 Planner , 2014 .

[75]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[76]  Luca Pulina,et al.  Applying Machine Learning Techniques to ASP Solving , 2012, ICLP.

[77]  Lothar Thiele,et al.  Defining and Optimizing Indicator-Based Diversity Measures in Multiobjective Search , 2010, PPSN.

[78]  Ben Paechter,et al.  A Lifelong Learning Hyper-heuristic Method for Bin Packing , 2015, Evolutionary Computation.

[79]  Edmund K. Burke,et al.  The Multi-Funnel Structure of TSP Fitness Landscapes: A Visual Exploration , 2015, Artificial Evolution.

[80]  Tad Hogg,et al.  An Economics Approach to Hard Computational Problems , 1997, Science.

[81]  Jano I van Hemert,et al.  Evolving combinatorial problem instances that are difficult to solve. , 2006, Evolutionary computation.

[82]  Andries Petrus Engelbrecht,et al.  Quantifying ruggedness of continuous landscapes using entropy , 2009, 2009 IEEE Congress on Evolutionary Computation.

[83]  Markus Wagner,et al.  Discrepancy-based evolutionary diversity optimization , 2018, GECCO.

[84]  Jano I. van Hemert,et al.  Understanding TSP Difficulty by Learning from Evolved Instances , 2010, LION.

[85]  L. Darrell Whitley,et al.  Improving an exact solver for the traveling salesman problem using partition crossover , 2017, GECCO.

[86]  Lawrence Carin,et al.  Sparse multinomial logistic regression: fast algorithms and generalization bounds , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[87]  L. Darrell Whitley,et al.  Building a better heuristic for the traveling salesman problem: combining edge assembly crossover and partition crossover , 2017, GECCO.

[88]  Luca Pulina,et al.  A self-adaptive multi-engine solver for quantified Boolean formulas , 2009, Constraints.

[89]  Michel Gendreau,et al.  Hyper-heuristics: a survey of the state of the art , 2013, J. Oper. Res. Soc..

[90]  William J. Cook,et al.  The Traveling Salesman Problem: A Computational Study , 2007 .

[91]  Alfonso Gerevini,et al.  Portfolio Methods for Optimal Planning: An Empirical Analysis , 2015, 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI).

[92]  Gabriela Ochoa,et al.  Additional Dimensions to the Study of Funnels in Combinatorial Landscapes , 2016, GECCO.

[93]  Mohamed Slimane,et al.  A Critical and Empirical Study of Epistasis Measures for Predicting GA Performances: A Summary , 1997, Artificial Evolution.

[94]  Michael Mitzenmacher,et al.  Detecting Novel Associations in Large Data Sets , 2011, Science.

[95]  W. Armstrong,et al.  Dynamic Algorithm Selection Using Reinforcement Learning , 2006, 2006 International Workshop on Integrating AI and Data Mining.

[96]  Tomoharu Nagao,et al.  Bag of local landscape features for fitness landscape analysis , 2016, Soft Comput..

[97]  Lior Rokach,et al.  Ensemble-based classifiers , 2010, Artificial Intelligence Review.

[98]  Toby Walsh,et al.  Morphing: Combining Structure and Randomness , 1999, AAAI/IAAI.

[99]  Michèle Sebag,et al.  Bi-population CMA-ES agorithms with surrogate models and line searches , 2013, GECCO.

[100]  Alfonso Gerevini,et al.  An Automatically Configurable Portfolio-based Planner with Macro-actions: PbP , 2009, ICAPS.

[101]  Sheila A. McIlraith,et al.  VARSAT: Integrating Novel Probabilistic Inference Techniques with DPLL Search , 2009, SAT.

[102]  Dong-il Seo,et al.  An Information-Theoretic Analysis on the Interactions of Variables in Combinatorial Optimization Problems , 2007, Evolutionary Computation.

[103]  Shigenobu Kobayashi,et al.  A Powerful Genetic Algorithm Using Edge Assembly Crossover for the Traveling Salesman Problem , 2013, INFORMS J. Comput..

[104]  Jürgen Schmidhuber,et al.  Algorithm Selection as a Bandit Problem with Unbounded Losses , 2008, LION.

[105]  Andries Petrus Engelbrecht,et al.  Characterising constrained continuous optimisation problems , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[106]  Kevin Leyton-Brown,et al.  OASC-2017: *Zilla Submission , 2017, OASC.

[107]  Jakob Bossek,et al.  smoof: Single- and Multi-Objective Optimization Test Functions , 2017, R J..

[108]  Cesare Tinelli,et al.  Handbook of Satisfiability , 2021, Handbook of Satisfiability.

[109]  Yuri Malitsky,et al.  Model-Based Genetic Algorithms for Algorithm Configuration , 2015, IJCAI.

[110]  Heike Trautmann,et al.  Parameterization of state-of-the-art performance indicators: a robustness study based on inexact TSP solvers , 2018, GECCO.

[111]  Hsuan-Tien Lin,et al.  One-sided Support Vector Regression for Multiclass Cost-sensitive Classification , 2010, ICML.

[112]  Luís Torgo,et al.  OpenML: A Collaborative Science Platform , 2013, ECML/PKDD.

[113]  Günter Rudolph,et al.  Evaluation of a Multi-Objective EA on Benchmark Instances for Dynamic Routing of a Vehicle , 2015, GECCO.

[114]  Lukás Chrpa,et al.  ASAP: An Automatic Algorithm Selection Approach for Planning , 2014, Int. J. Artif. Intell. Tools.

[115]  Mario A. Muñoz,et al.  Landscape characterization of numerical optimization problems using biased scattered data , 2012, 2012 IEEE Congress on Evolutionary Computation.

[116]  Marco Laumanns,et al.  Scalable Test Problems for Evolutionary Multiobjective Optimization , 2005, Evolutionary Multiobjective Optimization.

[117]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[118]  Constantin Halatsis,et al.  Measures of Intrinsic Hardness for Constraint Satisfaction Problem Instances , 2004, SOFSEM.

[119]  Andrew W. Moore,et al.  Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation , 1993, NIPS.

[120]  Hao Wang,et al.  Towards Analyzing Multimodality of Continuous Multiobjective Landscapes , 2016, PPSN.

[121]  Kurt Hornik,et al.  kernlab - An S4 Package for Kernel Methods in R , 2004 .

[122]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[123]  Michael Affenzeller,et al.  A Comprehensive Survey on Fitness Landscape Analysis , 2012, Recent Advances in Intelligent Engineering Systems.

[124]  Heike Trautmann,et al.  Towards Analyzing Multimodality of Multiobjective Landscapes , 2016, PPSN 2016.

[125]  Heike Trautmann,et al.  Multi-objective Performance Measurement: Alternatives to PAR10 and Expected Running Time , 2018, LION.

[126]  Kate Smith-Miles,et al.  Performance Analysis of Continuous Black-Box Optimization Algorithms via Footprints in Instance Space , 2016, Evolutionary Computation.

[127]  Matthias Carnein,et al.  Optimizing Data Stream Representation: An Extensive Survey on Stream Clustering Algorithms , 2019, Bus. Inf. Syst. Eng..

[128]  Wanru Gao,et al.  Feature-Based Diversity Optimization for Problem Instance Classification , 2015, Evolutionary Computation.

[129]  Phil Husbands,et al.  Fitness Landscapes and Evolvability , 2002, Evolutionary Computation.

[130]  Anne Auger,et al.  Benchmarking the local metamodel CMA-ES on the noiseless BBOB'2013 test bed , 2013, GECCO.

[131]  Bernd Bischl,et al.  A feature-based comparison of local search and the christofides algorithm for the travelling salesperson problem , 2013, FOGA XII '13.

[132]  Ivana Kruijff-Korbayová,et al.  A Portfolio Approach to Algorithm Selection , 2003, IJCAI.

[133]  Lars Kotthoff,et al.  The Algorithm Selection Competition Series 2015-17 , 2018, ArXiv.

[134]  Thomas Stützle,et al.  A Racing Algorithm for Configuring Metaheuristics , 2002, GECCO.

[135]  Marius Thomas Lindauer,et al.  claspfolio 2: Advances in Algorithm Selection for Answer Set Programming , 2014, Theory and Practice of Logic Programming.

[136]  Carlos M. Fonseca,et al.  The Attainment-Function Approach to Stochastic Multiobjective Optimizer Assessment and Comparison , 2010, Experimental Methods for the Analysis of Optimization Algorithms.

[137]  Bernd Bischl,et al.  Exploratory landscape analysis , 2011, GECCO '11.

[138]  G. T. Timmer,et al.  Stochastic global optimization methods part II: Multi level methods , 1987, Math. Program..

[139]  Saman K. Halgamuge,et al.  On the selection of fitness landscape analysis metrics for continuous optimization problems , 2014, 7th International Conference on Information and Automation for Sustainability.

[140]  Bart Naudts,et al.  Epistasis as a Basic Concept in Formal Landscape Analysis , 1997, ICGA.

[141]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[142]  Nikos Pelekis,et al.  An evaluation of data stream clustering algorithms , 2018, Stat. Anal. Data Min..

[143]  Bernd Bischl,et al.  Reinforcement Learning for Automatic Online Algorithm Selection - an Empirical Study , 2016, ITAT.

[144]  Ivan Serina,et al.  Planning Through Stochastic Local Search and Temporal Action Graphs in LPG , 2003, J. Artif. Intell. Res..

[145]  Yuri Malitsky,et al.  ISAC - Instance-Specific Algorithm Configuration , 2010, ECAI.

[146]  Michael T. Wolfinger,et al.  Barrier Trees of Degenerate Landscapes , 2002 .

[147]  Lukás Chrpa,et al.  An Automatic Algorithm Selection Approach for Planning , 2013, 2013 IEEE 25th International Conference on Tools with Artificial Intelligence.

[148]  Bernd Bischl,et al.  Algorithm selection based on exploratory landscape analysis and cost-sensitive learning , 2012, GECCO '12.

[149]  Cyril Fonlupt,et al.  A Bit-Wise Epistasis Measure for Binary Search Spaces , 1998, PPSN.

[150]  Bernd Bischl,et al.  Cell Mapping Techniques for Exploratory Landscape Analysis , 2014 .

[151]  Hugo Terashima-Marín,et al.  Lifelong Learning Selection Hyper-heuristics for Constraint Satisfaction Problems , 2015, MICAI.

[152]  R. Lyndon While,et al.  A review of multiobjective test problems and a scalable test problem toolkit , 2006, IEEE Transactions on Evolutionary Computation.

[153]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .

[154]  Keld Helsgaun,et al.  An effective implementation of the Lin-Kernighan traveling salesman heuristic , 2000, Eur. J. Oper. Res..

[155]  Diane J. Cook,et al.  Maximizing the Benefits of Parallel Search Using Machine Learning , 1997, AAAI/IAAI.

[156]  Anne Auger,et al.  COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..

[157]  Barry O'Sullivan,et al.  SNNAP: Solver-Based Nearest Neighbor for Algorithm Portfolios , 2013, ECML/PKDD.

[158]  Leslie Pérez Cáceres,et al.  The irace package: Iterated racing for automatic algorithm configuration , 2016 .

[159]  Lars Kotthoff,et al.  Open Algorithm Selection Challenge 2017: Setup and Scenarios , 2017, OASC.

[160]  I. Moser,et al.  Constraint Handling Guided by Landscape Analysis in Combinatorial and Continuous Search Spaces , 2019, Evolutionary Computation.

[161]  Bernd Bischl,et al.  Local Search and the Traveling Salesman Problem: A Feature-Based Characterization of Problem Hardness , 2012, LION.

[162]  Günter Rudolph,et al.  Contemporary Evolution Strategies , 1995, ECAL.

[163]  Qingfu Zhang,et al.  Multiobjective optimization Test Instances for the CEC 2009 Special Session and Competition , 2009 .

[164]  Alfonso Gerevini,et al.  PbP2: Automatic Configuration of a Portfolio-based Multi-Planner , 2011, ICAPS 2011.

[165]  Yuri Malitsky,et al.  Boosting Sequential Solver Portfolios: Knowledge Sharing and Accuracy Prediction , 2013, LION.

[166]  Derek Long,et al.  Plan Constraints and Preferences in PDDL3 , 2006 .

[167]  Yuri Malitsky,et al.  Algorithm Selection and Scheduling , 2011, CP.

[168]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[169]  Thomas G. Dietterich Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[170]  Werner Ebeling,et al.  The Density of States - A Measure of the Difficulty of Optimisation Problems , 1996, PPSN.

[171]  Matthias Carnein,et al.  An Empirical Comparison of Stream Clustering Algorithms , 2017, Conf. Computing Frontiers.

[172]  Tim Jones Evolutionary Algorithms, Fitness Landscapes and Search , 1995 .

[173]  Andries Petrus Engelbrecht,et al.  A survey of techniques for characterising fitness landscapes and some possible ways forward , 2013, Inf. Sci..

[174]  Shigenobu Kobayashi,et al.  Edge Assembly Crossover: A High-Power Genetic Algorithm for the Travelling Salesman Problem , 1997, ICGA.

[175]  Anne Auger,et al.  COCO: The Bi-objective Black Box Optimization Benchmarking (bbob-biobj) Test Suite , 2016, ArXiv.

[176]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[177]  Stefan Edelkamp,et al.  Automated Planning: Theory and Practice , 2007, Künstliche Intell..

[178]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[179]  Fred W. Glover,et al.  Scatter Search and Local Nlp Solvers: A Multistart Framework for Global Optimization , 2006, INFORMS J. Comput..

[180]  R. Geoff Dromey,et al.  An algorithm for the selection problem , 1986, Softw. Pract. Exp..

[181]  Xiaodong Li,et al.  Benchmark Functions for CEC'2013 Special Session and Competition on Niching Methods for Multimodal Function Optimization' , 2013 .

[182]  Pascal Kerschke,et al.  An Expedition to Multimodal Multi-objective Optimization Landscapes , 2017, EMO.

[183]  Geoff Holmes,et al.  The online performance estimation framework: heterogeneous ensemble learning for data streams , 2017, Machine Learning.

[184]  Bernd Bischl,et al.  ASlib: A benchmark library for algorithm selection , 2015, Artif. Intell..

[185]  Frank Neumann,et al.  A Feature-Based Comparison of Evolutionary Computing Techniques for Constrained Continuous Optimisation , 2015, ICONIP.

[186]  Yuval Davidor,et al.  Epistasis Variance: A Viewpoint on GA-Hardness , 1990, FOGA.

[187]  Joseph C. Culberson,et al.  On the Futility of Blind Search: An Algorithmic View of No Free Lunch , 1998, Evolutionary Computation.

[188]  Jano I. van Hemert,et al.  Discovering the suitability of optimisation algorithms by learning from evolved instances , 2011, Annals of Mathematics and Artificial Intelligence.

[189]  Heike Trautmann,et al.  The R-Package FLACCO for exploratory landscape analysis with applications to multi-objective optimization problems , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[190]  Katharina Eggensperger,et al.  Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters , 2013 .

[191]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[192]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Selection of algorithms to solve traveling salesman problems using meta-learning , 2011, Int. J. Hybrid Intell. Syst..

[193]  Mario A. Muñoz,et al.  The Algorithm Selection Problem on the Continuous Optimization Domain , 2013 .

[194]  Keld Helsgaun,et al.  General k-opt submoves for the Lin–Kernighan TSP heuristic , 2009, Math. Program. Comput..

[195]  Marie desJardins,et al.  What Makes Planners Predictable? , 2008, ICAPS.

[196]  Marius Thomas Lindauer,et al.  From Sequential Algorithm Selection to Parallel Portfolio Selection , 2015, LION.

[197]  Heike Trautmann,et al.  Detecting Funnel Structures by Means of Exploratory Landscape Analysis , 2015, GECCO.

[198]  Heike Trautmann,et al.  Automated and Feature-Based Problem Characterization and Algorithm Selection Through Machine Learning , 2018 .

[199]  Alfonso Gerevini,et al.  Portfolio Methods for Optimal Planning: An Empirical Analysis , 2015, 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI).

[200]  Olivier Roussel,et al.  The International SAT Solver Competitions , 2012, AI Mag..

[201]  Kevin Leyton-Brown,et al.  Algorithm runtime prediction: Methods & evaluation , 2012, Artif. Intell..

[202]  Kevin Leyton-Brown,et al.  An evaluation of sequential model-based optimization for expensive blackbox functions , 2013, GECCO.

[203]  Alex Fukunaga,et al.  Genetic algorithm portfolios , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[204]  Saman K. Halgamuge,et al.  Quantifying Variable Interactions in Continuous Optimization Problems , 2017, IEEE Transactions on Evolutionary Computation.

[205]  Mario A. Muñoz,et al.  Algorithm selection for black-box continuous optimization problems: A survey on methods and challenges , 2015, Inf. Sci..

[206]  Kenneth Alan De Jong,et al.  An analysis of the behavior of a class of genetic adaptive systems. , 1975 .

[207]  Heike Trautmann,et al.  Low-Budget Exploratory Landscape Analysis on Multiple Peaks Models , 2016, GECCO.

[208]  Pascal Kerschke,et al.  flaccogui: exploratory landscape analysis for everyone , 2017, GECCO.

[209]  Kai Ming Ting,et al.  An Instance-weighting Method to Induce Cost-sensitive Trees , 2001 .

[210]  Kevin Leyton-Brown,et al.  Improved Features for Runtime Prediction of Domain-Independent Planners , 2014, ICAPS.

[211]  Mike Preuss,et al.  Multimodal Optimization by Means of Evolutionary Algorithms , 2015, Natural Computing Series.

[212]  Günter Rudolph,et al.  Local search effects in bi-objective orienteering , 2018, GECCO.

[213]  Kate Smith-Miles,et al.  Instance spaces for machine learning classification , 2017, Machine Learning.

[214]  Anne Auger,et al.  Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions , 2009 .

[215]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[216]  Kalyanmoy Deb,et al.  Constraint handling in efficient global optimization , 2017, GECCO.

[217]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[218]  Gary B. Lamont,et al.  Multiobjective evolutionary algorithms: classifications, analyses, and new innovations , 1999 .

[219]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[220]  Carlos M. Fonseca,et al.  Multiobjective genetic algorithms with application to control engineering problems. , 1995 .

[221]  Sharad Malik,et al.  Zchaff2004: An Efficient SAT Solver , 2004, SAT (Selected Papers.

[222]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.