论文信息 - Benchmarking in Optimization: Best Practice and Open Issues

Benchmarking in Optimization: Best Practice and Open Issues

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.

[1] Ranjit K. Roy,et al. Design of Experiments Using The Taguchi Approach: 16 Steps to Product and Process Improvement , 2001 .

[2] Thomas Bartz-Beielstein,et al. Experimental Methods for the Analysis of Optimization Algorithms , 2010 .

[3] Markus Wagner,et al. Better software analytics via “DUO”: Data mining algorithms using/used-by optimizers , 2020, Empirical Software Engineering.

[4] Matthias Müller-Hannemann,et al. Algorithm Engineering: Bridging the Gap between Algorithm Theory and Practice [outcome of a Dagstuhl Seminar] , 2010, Algorithm Engineering.

[5] Heike Trautmann,et al. A multi-objective perspective on performance assessment and automated selection of single-objective optimization algorithms , 2020, Appl. Soft Comput..

[6] Thomas Bartz-Beielstein,et al. Improving the reliability of test functions generators , 2020, Appl. Soft Comput..

[7] Stuart A. Kauffman,et al. The origins of order , 1993 .

[8] Markus Wagner,et al. Evolutionary many-objective optimization: A quick-start guide , 2015 .

[9] L. Darrell Whitley,et al. Evaluating Evolutionary Algorithms , 1996, Artif. Intell..

[10] Thomas Bartz-Beielstein,et al. Model-based methods for continuous and discrete global optimization , 2017, Appl. Soft Comput..

[11] Kevin Leyton-Brown,et al. Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[12] Yuri Malitsky,et al. Model-Based Genetic Algorithms for Algorithm Configuration , 2015, IJCAI.

[13] Eckart Zitzler,et al. Evolutionary algorithms for multiobjective optimization: methods and applications , 1999 .

[14] David S. Johnson,et al. A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[15] Thomas J. Santner,et al. Design and analysis of computer experiments , 1998 .

[16] Jürgen Branke,et al. Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[17] Zbigniew Michalewicz,et al. Benchmarking Optimization Algorithms: An Open Source Framework for the Traveling Salesman Problem , 2014, IEEE Computational Intelligence Magazine.

[18] S. Shapiro,et al. An Analysis of Variance Test for Normality (Complete Samples) , 1965 .

[19] Heike Trautmann,et al. Parameterization of state-of-the-art performance indicators: a robustness study based on inexact TSP solvers , 2018, GECCO.

[20] Daan van den Berg,et al. Fireworks Algorithm versus Plant Propagation Algorithm , 2019, IJCCI.

[21] Maliha S. Nash,et al. Handbook of Parametric and Nonparametric Statistical Procedures , 2001, Technometrics.

[22] Benjamín Barán,et al. Performance metrics in multi-objective optimization , 2015, 2015 Latin American Computing Conference (CLEI).

[23] Manuela Herman,et al. Design Of Experiments Using The Taguchi Approach 16 Steps To Product And Process Improvement , 2016 .

[24] K. Popper. Objective Knowledge: An Evolutionary Approach , 1972 .

[25] Heike Trautmann,et al. Benchmarking Evolutionary Algorithms: Towards Exploratory Landscape Analysis , 2010, PPSN.

[26] Xin Yao,et al. Dynamic Multi-objective Optimization: A Survey of the State-of-the-Art , 2013 .

[27] Sébastien Vérel,et al. Dominance, epsilon, and hypervolume local optimal sets in multi-objective optimization, and how to tell the difference , 2018, GECCO.

[28] Anne Auger,et al. COCO: a platform for comparing continuous optimizers in a black-box setting , 2016, Optim. Methods Softw..

[29] Mark Hoogendoorn,et al. Parameter Control in Evolutionary Algorithms: Trends and Challenges , 2015, IEEE Transactions on Evolutionary Computation.

[30] Donald R. Jones,et al. Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[31] Zbigniew Michalewicz,et al. A comprehensive benchmark set and heuristics for the traveling thief problem , 2014, GECCO.

[32] Hans-Paul Schwefel,et al. Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[33] Andries Petrus Engelbrecht,et al. A survey of techniques for characterising fitness landscapes and some possible ways forward , 2013, Inf. Sci..

[34] Dimo Brockhoff,et al. Mixed-integer benchmark problems for single- and bi-objective optimization , 2019, GECCO.

[35] Thomas Bartz-Beielstein,et al. Experimental Research in Evolutionary Computation - The New Experimentalism , 2010, Natural Computing Series.

[36] El-Ghazali Talbi,et al. Metaheuristics - From Design to Implementation , 2009 .

[37] Talal Rahwan,et al. Using the Shapley Value to Analyze Algorithm Portfolios , 2016, AAAI.

[38] F. Pesarin. Multivariate Permutation Tests : With Applications in Biostatistics , 2001 .

[39] Panos M. Pardalos,et al. No Free Lunch Theorem: A Review , 2019, Approximation and Optimization.

[40] Marjan Mernik,et al. A chess rating system for evolutionary algorithms: A new method for the comparison and ranking of evolutionary algorithms , 2014, Inf. Sci..

[41] Mathijs de Weerdt,et al. Black-box Mixed-Variable Optimisation using a Surrogate Model that Satisfies Integer Constraints , 2020, ArXiv.

[42] M. Preuss,et al. Search Dynamics on Multimodal Multiobjective Problems , 2019, Evolutionary Computation.

[43] D. Mayo,et al. Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction , 2006, The British Journal for the Philosophy of Science.

[44] David H. Wolpert,et al. No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[45] Olivier Teytaud,et al. Critical Hyper-Parameters: No Random, No Cry , 2017, ArXiv.

[46] Thomas Stützle,et al. Ant colony optimization: artificial ants as a computational intelligence technique , 2006 .

[47] Kevin Leyton-Brown,et al. SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[48] Heike Trautmann,et al. Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning , 2017, Evolutionary Computation.

[49] Elizabeth F. Wanner,et al. A Multicriteria Statistical Based Comparison Methodology for Evaluating Evolutionary Algorithms , 2011, IEEE Transactions on Evolutionary Computation.

[50] Hisao Ishibuchi,et al. An easy-to-use real-world multi-objective optimization problem suite , 2020, Appl. Soft Comput..

[51] Heike Trautmann,et al. Anytime Behavior of Inexact TSP Solvers and Perspectives for Automated Algorithm Selection , 2020, 2020 IEEE Congress on Evolutionary Computation (CEC).

[52] Dirk Thierens,et al. The balance between proximity and diversity in multiobjective evolutionary algorithms , 2003, IEEE Trans. Evol. Comput..

[53] Jürgen Branke,et al. Evolutionary Optimization in Dynamic Environments , 2001, Genetic Algorithms and Evolutionary Computation.

[54] Tome Eftimov,et al. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis , 2020, Appl. Soft Comput..

[55] Mark S. Boddy,et al. An Analysis of Time-Dependent Planning , 1988, AAAI.

[56] Philip J. Fleming,et al. How not to lie with statistics: the correct way to summarize benchmark results , 1986, CACM.

[57] Fred Glover,et al. Tabu Search - Part II , 1989, INFORMS J. Comput..

[58] Zbigniew Michalewicz,et al. The travelling thief problem: The first step in the transition from theoretical problems to realistic problems , 2013, 2013 IEEE Congress on Evolutionary Computation.

[59] K. Steiglitz,et al. Adaptive step size random search , 1968 .

[60] H. Levene. Robust tests for equality of variances , 1961 .

[61] Hao Wang,et al. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics , 2018, ArXiv.

[62] Thomas Bäck,et al. Mixed-Integer NK Landscapes , 2006, PPSN.

[63] Raphael T. Haftka,et al. Requirements for papers focusing on new or improved global optimization algorithms , 2016 .

[64] Thomas Bäck,et al. IOHanalyzer: Performance Analysis for Iterative Optimization Heuristic , 2020, ArXiv.

[65] John N. Hooker,et al. Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[66] Thomas Bartz-Beielstein,et al. Sequential parameter optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[67] Markus Wagner,et al. Evolutionary diversity optimization using multi-objective indicators , 2018, GECCO.

[68] H. H. Rosenbrock,et al. An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[69] R. Swinburne. OBJECTIVE KNOWLEDGE: AN EVOLUTIONARY APPROACH , 1973 .

[70] Jason H. Moore,et al. Where are we now?: a large benchmark study of recent symbolic regression methods , 2018, GECCO.

[71] Thomas Bartz-Beielstein,et al. Design and Analysis of Optimization Algorithms Using Computational Statistics , 2004 .

[72] R. Plackett,et al. THE DESIGN OF OPTIMUM MULTIFACTORIAL EXPERIMENTS , 1946 .

[73] Lars Kotthoff,et al. Automated Machine Learning: Methods, Systems, Challenges , 2019, The Springer Series on Challenges in Machine Learning.

[74] F. Pillichshammer,et al. Digital Nets and Sequences: Discrepancy Theory and Quasi-Monte Carlo Integration , 2010 .

[75] Zbigniew Michalewicz,et al. Evolutionary computation for multicomponent problems: opportunities and future directions , 2016, Optimization in Industry.

[76] Tome Eftimov,et al. A Novel Approach to statistical comparison of meta-heuristic stochastic optimization algorithms using deep statistics , 2017, Inf. Sci..

[77] Bernd Bischl,et al. ASlib: A benchmark library for algorithm selection , 2015, Artif. Intell..

[78] Olivier Teytaud,et al. Noisy Optimization: Fast Convergence Rates with Comparison-Based Algorithms , 2016, GECCO.

[79] Frank Neumann,et al. Bioinspired computation in combinatorial optimization: algorithms and their computational complexity , 2010, GECCO '12.

[80] Bernd Bischl,et al. Exploratory landscape analysis , 2011, GECCO '11.

[81] Gerhard W. Dueck,et al. Threshold accepting: a general purpose optimization algorithm appearing superior to simulated anneal , 1990 .

[82] Riccardo Poli,et al. Particle swarm optimization , 1995, Swarm Intelligence.

[83] Tome Eftimov,et al. DSCTool: A web-service-based framework for statistical comparison of stochastic optimization algorithms , 2020, Appl. Soft Comput..

[84] Francisco Herrera,et al. A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization , 2009, J. Heuristics.

[85] W. Kruskal,et al. Use of Ranks in One-Criterion Variance Analysis , 1952 .

[86] Ehl Emile Aarts,et al. Simulated annealing and Boltzmann machines , 2003 .

[87] Francisco Herrera,et al. Analyzing convergence performance of evolutionary algorithms: A statistical approach , 2014, Inf. Sci..

[88] Shlomo Zilberstein,et al. Using Anytime Algorithms in Intelligent Systems , 1996, AI Mag..

[89] A. E. Eiben,et al. Introduction to Evolutionary Computing , 2003, Natural Computing Series.

[90] Petros Koumoutsakos,et al. Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[91] Patrick M. Reed,et al. Diagnostic Assessment of Search Controls and Failure Modes in Many-Objective Evolutionary Optimization , 2012, Evolutionary Computation.

[92] L. Huson,et al. Multivariate Permutation Tests: with Applications in Biostatistics , 2003 .

[93] Toby Walsh,et al. How Not To Do It , 1995 .

[94] K. Price. Differential evolution vs. the functions of the 2/sup nd/ ICEO , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[95] Warren Hare,et al. Best practices for comparing optimization algorithms , 2017, Optimization and Engineering.

[96] Thomas Stützle,et al. A Racing Algorithm for Configuring Metaheuristics , 2002, GECCO.

[97] Marcus Gallagher,et al. Towards improved benchmarking of black-box optimization algorithms using clustering problems , 2016, Soft Comput..

[98] Geoffrey E. Hinton,et al. Visualizing Data using t-SNE , 2008 .

[99] Jean-Paul Watson,et al. Testing, Evaluation and Performance of Optimization and Learning Systems , 2002 .

[100] Jorge J. Moré,et al. Testing Unconstrained Optimization Software , 1981, TOMS.

[101] Kate Smith-Miles,et al. Generating new test instances by evolving in instance space , 2015, Comput. Oper. Res..

[102] Stephan Mertens,et al. Low autocorrelation binary sequences , 2015, 1512.02475.

[103] Margaret J. Robertson,et al. Design and Analysis of Experiments , 2006, Handbook of statistics.

[104] Anne Auger,et al. Using Well-Understood Single-Objective Functions in Multiobjective Black-Box Optimization Test Suites , 2019, Evolutionary Computation.

[105] Tome Eftimov,et al. Identifying practical significance through statistical comparison of meta-heuristic stochastic optimization algorithms , 2019, Appl. Soft Comput..

[106] Daan van den Berg,et al. Parameter Sensitivity Patterns in the Plant Propagation Algorithm , 2020, IJCCI.

[107] Thomas Stützle,et al. Classification of Metaheuristics and Design of Experiments for the Analysis of Components , 2001 .

[108] Thomas Stützle,et al. Ant Colony Optimization for Mixed-Variable Optimization Problems , 2014, IEEE Transactions on Evolutionary Computation.

[109] Thomas Bartz-Beielstein,et al. In a Nutshell: Sequential Parameter Optimization , 2017, ArXiv.

[110] Heder S. Bernardino,et al. A benchmark suite for designing combinational logic circuits via metaheuristics , 2020, Appl. Soft Comput..

[111] O. Teytaud,et al. Fully Parallel Hyperparameter Search: Reshaped Space-Filling , 2020, ICML.

[112] Brian A. Nosek,et al. The preregistration revolution , 2018, Proceedings of the National Academy of Sciences.

[113] Stefan M. Wild,et al. Benchmarking Derivative-Free Optimization Algorithms , 2009, SIAM J. Optim..

[114] Francisco J. Rodríguez,et al. Arbitrary function optimisation with metaheuristics , 2012, Soft Comput..

[115] Cecilia R. Aragon,et al. Optimization by Simulated Annealing: An Experimental Evaluation; Part II, Graph Coloring and Number Partitioning , 1991, Oper. Res..

[116] Wei Chen,et al. Paradoxes in Numerical Comparison of Optimization Algorithms , 2020, IEEE Transactions on Evolutionary Computation.

[117] Michael Jackson,et al. Optimal Design of Experiments , 1994 .

[118] Markus Wagner,et al. On the Effectiveness of Simple Success-Based Parameter Selection Mechanisms for Two Classical Discrete Black-Box Optimization Benchmark Problems , 2018, ArXiv.

[119] Randal S. Olson,et al. TPOT: A Tree-based Pipeline Optimization Tool for Automating Machine Learning , 2016, AutoML@ICML.

[120] Pascal Kerschke,et al. Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-Package Flacco , 2017, Studies in Classification, Data Analysis, and Knowledge Organization.

[121] David D. Cox,et al. Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms , 2013, SciPy.

[122] Richard J. Beckman,et al. A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[123] Thomas Reinartz,et al. CRISP-DM 1.0: Step-by-step data mining guide , 2000 .

[124] M. Friedman. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance , 1937 .

[125] Ofer M. Shir,et al. Compiling a benchmarking test-suite for combinatorial black-box optimization: a position paper , 2018, GECCO.

[126] Koji Tsuda,et al. COMBO: An efficient Bayesian optimization library for materials science , 2016 .

[127] Shengxiang Yang,et al. Evolutionary dynamic optimization: A survey of the state of the art , 2012, Swarm Evol. Comput..

[128] Jürgen Branke,et al. Efficient search for robust solutions by means of evolutionary algorithms and fitness approximation , 2006, IEEE Transactions on Evolutionary Computation.

[129] Thomas Bäck,et al. Theory of Evolutionary Computation: Recent Developments in Discrete Optimization , 2020, Theory of Evolutionary Computation.

[130] Guigang Zhang,et al. Deep Learning , 2016, Int. J. Semantic Comput..

[131] Xin Yao,et al. Many-Objective Evolutionary Algorithms , 2015, ACM Comput. Surv..

[132] Randal S. Olson,et al. PMLB: a large benchmark suite for machine learning evaluation and comparison , 2017, BioData Mining.

[133] Jack P. C. Kleijnen. Design and Analysis of Simulation Experiments , 2007 .

[134] Bernd Bischl,et al. A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem , 2012, Annals of Mathematics and Artificial Intelligence.

[135] Jeffrey Horn,et al. Handbook of evolutionary computation , 1997 .

[136] Patrick Siarry,et al. A survey on optimization metaheuristics , 2013, Inf. Sci..

[137] Thomas Stützle,et al. AClib: A Benchmark Library for Algorithm Configuration , 2014, LION.

[138] Marco Dorigo,et al. Ant colony optimization for continuous domains , 2008, Eur. J. Oper. Res..

[139] Nikolaus Hansen,et al. Invariance, Self-Adaptation and Correlated Mutations and Evolution Strategies , 2000, PPSN.

[140] T. W. Anderson,et al. Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes , 1952 .

[141] Pascal Kerschke,et al. Single- and multi-objective game-benchmark for evolutionary algorithms , 2019, GECCO.

[142] Bilel Derbel,et al. Algorithm selection of anytime algorithms , 2020, GECCO.

[143] Joshua D. Knowles,et al. On metrics for comparing nondominated sets , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[144] R. Lyndon While,et al. A review of multiobjective test problems and a scalable test problem toolkit , 2006, IEEE Transactions on Evolutionary Computation.

[145] Andries Petrus Engelbrecht,et al. Analysis and classification of optimisation benchmark functions and benchmark suites , 2014, 2014 IEEE Congress on Evolutionary Computation (CEC).

[146] Thiago Santos,et al. A Convergence indicator for Multi-Objective Optimisation Algorithms , 2018, TEMA (São Carlos).

[147] David J. Sheskin,et al. Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[148] P. Suganthan,et al. Problem Definitions and Evaluation Criteria for the CEC 2010 Competition on Constrained Real- Parameter Optimization , 2010 .

[149] Fred W. Glover,et al. Tabu Search - Part I , 1989, INFORMS J. Comput..

[150] Jorge J. Moré,et al. Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[151] Abdullah Al Mamun,et al. Evolutionary big optimization (BigOpt) of signals , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[152] Mark S. Boddy,et al. Solving Time-Dependent Planning Problems , 1989, IJCAI.

[153] Robert E. Shannon,et al. Design and analysis of simulation experiments , 1978, WSC '78.

[154] M. Hansen,et al. Evaluating the quality of approximations to the non-dominated set , 1998 .

[155] David E. Goldberg,et al. Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[156] Richard S. Barr,et al. Network Reoptimization Algorithms: A Statistically Designed Comparison , 1993, INFORMS J. Comput..

[157] Markus Wagner,et al. Towards rigorous validation of energy optimisation experiments , 2020, GECCO.

[158] Benjamin Doerr,et al. Self-Adjusting Mutation Rates with Provably Optimal Success Rules , 2019, Algorithmica.

[159] Dirk V. Arnold,et al. Noisy Optimization With Evolution Strategies , 2002, Genetic Algorithms and Evolutionary Computation.

[160] Cecilia R. Aragon,et al. Optimization by Simulated Annealing: An Experimental Evaluation; Part I, Graph Partitioning , 1989, Oper. Res..

[161] H. Lindman. Analysis of variance in complex experimental designs , 1974 .

[162] Leslie Pérez Cáceres,et al. The irace package: Iterated racing for automatic algorithm configuration , 2016 .

[163] Kenneth de Jong. Parameter Setting in EAs: a 30 Year Perspective , 2007 .

[164] Shigeyoshi Tsutsui,et al. A Robust Solution Searching Scheme in Genetic Search , 1996, PPSN.

[165] M. Bartlett. Properties of Sufficiency and Statistical Tests , 1992 .

[166] James McDermott,et al. When and why metaheuristics researchers can ignore “No Free Lunch” theorems , 2019, ArXiv.

[167] Mansooreh Mollaghasemi,et al. Using genetic algorithms and an indifference-zone ranking and selection procedure under common random numbers for simulation optimisation , 2012, J. Simulation.

[168] Marjan Mernik,et al. Replication and comparison of computational experiments in applied evolutionary computing: Common pitfalls and guidelines to avoid them , 2014, Appl. Soft Comput..

[169] Jack P. C. Kleijnen,et al. Regression and Kriging metamodels with their experimental designs in simulation: A review , 2017, Eur. J. Oper. Res..

[170] Alastair Smith,et al. How not to do it , 2005 .

[171] Pierre Hansen,et al. Variable Neighborhood Search , 2018, Handbook of Heuristics.

[172] Catherine C. McGeoch. Feature Article - Toward an Experimental Method for Algorithm Simulation , 1996, INFORMS J. Comput..

[173] Kate Smith-Miles,et al. Measuring algorithm footprints in instance space , 2012, 2012 IEEE Congress on Evolutionary Computation.

[174] Anne Auger,et al. Performance evaluation of an advanced local search evolutionary algorithm , 2005, 2005 IEEE Congress on Evolutionary Computation.

[175] F. Wilcoxon. Individual Comparisons by Ranking Methods , 1945 .

[176] Jürgen Branke,et al. Creating Robust Solutions by Means of Evolutionary Algorithms , 1998, PPSN.

[177] J. Neyman. Contributions to the theory of statistics , 1956 .

[178] Ofer M. Shir,et al. Benchmarking discrete optimization heuristics with IOHprofiler , 2019, GECCO.

[179] Kate Smith-Miles,et al. Performance Analysis of Continuous Black-Box Optimization Algorithms via Footprints in Instance Space , 2016, Evolutionary Computation.

[180] John W. Tukey,et al. Exploratory Data Analysis. , 1979 .

[181] Jason H. Moore,et al. Benchmarking Manifold Learning Methods on a Large Collection of Datasets , 2020, EuroGP.

[182] Jürgen Branke,et al. Efficient fitness estimation in noisy environments , 2001 .

[183] Kenneth DeJong,et al. Parameter Setting in EAs: a 30 Year Perspective , 2007, Parameter Setting in Evolutionary Algorithms.

[184] Anne Auger,et al. Theory of Randomized Search Heuristics , 2012, Algorithmica.

[185] Per Kristian Lehre,et al. Black-Box Search by Unbiased Variation , 2010, GECCO '10.

[186] C. D. Gelatt,et al. Optimization by Simulated Annealing , 1983, Science.

[187] Joseph C. Culberson,et al. On the Futility of Blind Search: An Algorithmic View of No Free Lunch , 1998, Evolutionary Computation.

[188] J. A. Lozano,et al. Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[189] Thomas Bartz-Beielstein,et al. A new Taxonomy of Continuous Global Optimization Algorithms , 2018, ArXiv.

[190] Katharina Eggensperger,et al. Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters , 2013 .

[191] Francisco Herrera,et al. A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms , 2011, Swarm Evol. Comput..

[192] Bruce L. Golden,et al. Experimentation in optimization , 1986 .

[193] Kalyanmoy Deb,et al. Investigating the Effect of Imbalance Between Convergence and Diversity in Evolutionary Multiobjective Algorithms , 2017, IEEE Transactions on Evolutionary Computation.

[194] Qiang Yang,et al. A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[195] Thomas Weise,et al. Difficult features of combinatorial optimization problems and the tunable w-model benchmark problem for simulating them , 2018, GECCO.

[196] Manuel López-Ibáñez,et al. Ant colony optimization , 2010, GECCO '10.

[197] John N. Hooker,et al. Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[198] Pablo San Segundo,et al. Research trends in combinatorial optimization , 2020, Int. Trans. Oper. Res..

[199] Qingfu Zhang,et al. Approximating the Set of Pareto-Optimal Solutions in Both the Decision and Objective Spaces by an Estimation of Distribution Algorithm , 2009, IEEE Transactions on Evolutionary Computation.

[200] Gary James Jason,et al. The Logic of Scientific Discovery , 1988 .

[201] Sonja Kuhnt,et al. Design and analysis of computer experiments , 2010 .

[202] Kent McClymont,et al. Benchmark multi-objective optimisation test problems with mixed encodings , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[203] Tome Eftimov,et al. The impact of statistics for benchmarking in evolutionary computation research , 2018, GECCO.

[204] Doug Hains,et al. Hyperplane initialized local search for MAXSAT , 2013, GECCO '13.

[205] Mauricio G. C. Resende,et al. Designing and reporting on computational experiments with heuristic methods , 1995, J. Heuristics.

[206] Olivier Teytaud,et al. Variance Reduction for Better Sampling in Continuous Domains , 2020, PPSN.

[207] Eric R. Ziegel,et al. Probability and Statistics for Engineering and the Sciences , 2004, Technometrics.

[208] Markus Wagner,et al. Evolving diverse TSP instances by means of novel and creative mutation operators , 2019, FOGA '19.

[209] Takuya Akiba,et al. Optuna: A Next-generation Hyperparameter Optimization Framework , 2019, KDD.

[210] A. E. Eiben,et al. Evolutionary Algorithm Parameters and Methods to Tune Them , 2012, Autonomous Search.

[211] Heike Trautmann,et al. Automated Algorithm Selection: Survey and Perspectives , 2018, Evolutionary Computation.

[212] J. Kleijnen. Analyzing simulation experiments with common random numbers , 1988 .

[213] N. Hansen,et al. Real-Parameter Black-Box Optimization Benchmarking: Experimental Setup , 2010 .

[214] H. Mühlenbein,et al. From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[215] Ameet Talwalkar,et al. Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization , 2016, J. Mach. Learn. Res..

[216] Thomas Stützle,et al. Pareto Local Optimum Sets in the Biobjective Traveling Salesman Problem: An Experimental Study , 2004, Metaheuristics for Multiobjective Optimisation.

[217] Anna Hart,et al. Mann-Whitney test is not just a test of medians: differences in spread can be important , 2001, BMJ : British Medical Journal.

[218] M. Braga,et al. Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[219] Yue Shi,et al. A modified particle swarm optimizer , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[220] ATSPDavid S. JohnsonAT. Experimental Analysis of Heuristics for the Stsp , 2001 .

[221] Alessandro Carosio,et al. Eidgenössische Technische Hochschule in Zürich , 2006, Schweizerische Zeitschrift für Hydrologie.

[222] R. Fletcher. Conjugate gradient methods for indefinite systems , 1976 .

[223] A. E. Eiben,et al. A critical note on experimental research methodology in EC , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[224] Michael D. Vose,et al. Unbiased black box search algorithms , 2011, GECCO '11.

[225] Markus Wagner,et al. The Dynamic Travelling Thief Problem: Benchmarks and Performance of Evolutionary Algorithms , 2020, ICONIP.

[226] Carlos A. Coello Coello,et al. A Study of the Parallelization of a Coevolutionary Multi-objective Evolutionary Algorithm , 2004, MICAI.

[227] Richard Craig Van Nostrand,et al. Design of Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement , 2002, Technometrics.

[228] Olivier Teytaud,et al. Versatile black-box optimization , 2020, GECCO.

[229] Michèle Sebag,et al. Analyzing bandit-based adaptive operator selection mechanisms , 2010, Annals of Mathematics and Artificial Intelligence.

[230] John M. Mulvey,et al. On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[231] Joerg joke Heitkoetter,et al. The hitch-hiker''s guide to evolutionary computation , 2001 .

[232] Mike Preuss. Experimentation in Evolutionary Computation , 2015 .

[233] Raymond Ros,et al. Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup , 2009 .

[234] Ivan Reinaldo Meneghini,et al. Scalable and Customizable Benchmark Problems for Many-Objective Optimization , 2020, Appl. Soft Comput..

[235] Thomas Stützle,et al. An Empirical Assessment of the Properties of Inverted Generational Distance on Multi- and Many-Objective Optimization , 2017, EMO.