ParamILS: an automatic algorithm configuration framework

The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More formally, we provide methods for optimizing a target algorithm's performance on a given class of problem instances by varying a set of ordinal and/or categorical parameters. We review a family of local-search-based algorithm configuration procedures and present novel techniques for accelerating them by adaptively limiting the time spent for evaluating individual configurations. We describe the results of a comprehensive experimental evaluation of our methods, based on the configuration of prominent complete and incomplete algorithms for SAT. We also present what is, to our knowledge, the first published work on automatically configuring the CPLEX mixed integer programming solver. All the algorithms we considered had default parameter settings that were manually identified with considerable effort. Nevertheless, using our automated algorithm configuration procedures, we achieved substantial and consistent performance improvements.

[1]  Hector J. Levesque,et al.  A New Method for Solving Hard Satisfiability Problems , 1992, AAAI.

[2]  Gerald DeJong,et al.  COMPOSER: A Probabilistic Solution to the Utility Problem in Speed-Up Learning , 1992, AAAI.

[3]  Steven Minton,et al.  Minimizing Conflicts: A Heuristic Repair Method for Constraint Satisfaction and Scheduling Problems , 1992, Artif. Intell..

[4]  Andrew W. Moore,et al.  Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation , 1993, NIPS.

[5]  Steven Minton.  An Analytic Learning System for Specializing Heuristics , 1993, IJCAI.

[6]  Jonathan Gratch,et al.  Adaptive Problem-solving for Large-scale Scheduling Problems: A Case Study , 1996, J. Artif. Intell. Res..

[7]  Bart Selman,et al.  Problem Structure in the Presence of Perturbations , 1997, AAAI/IAAI.

[8]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[9]  R. Dechter,et al.  Stochastic Local Search for Bayesian Networks , 1999 .

[10]  David S. Johnson,et al.  A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[11]  Rina Dechter,et al.  Stochastic local search for Bayesian network , 1999, AISTATS.

[12]  H. Terashima-Marín,et al.  Evolution of Constraint Satisfaction strategies in examination timetabling , 1999 .

[13]  Toby Walsh,et al.  Morphing: Combining Structure and Randomness , 1999, AAAI/IAAI.

[14]  Yoav Shoham,et al.  Towards a universal test suite for combinatorial auction algorithms , 2000, EC '00.

[15]  George C. Runger,et al.  Using Experimental Design to Find Effective Parameter Settings for Heuristics , 2001, J. Heuristics.

[16]  M. Dorigo,et al.  Design of Iterated Local Search Algorithms An Example Application to the Single Machine Total Weighted Tardiness Problem , 2001 .

[17]  Matthijs den Besten,et al.  Design of Iterated Local Search Algorithms , 2001, EvoWorkshops.

[18]  David Maxwell Chickering,et al.  A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report) , 2001, Electron. Notes Discret. Math..

[19]  Helena Ramalhinho Dias Lourenço,et al.  Iterated Local Search , 2001, Handbook of Metaheuristics.

[20]  Henry A. Kautz,et al.  Auto-Walksat: A Self-Tuning Implementation of Walksat , 2001, Electron. Notes Discret. Math..

[21]  Holger H. Hoos,et al.  Scaling and Probabilistic Smoothing: Efficient Dynamic Local Search for SAT , 2002, CP.

[22]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[23]  Thomas Stützle,et al.  A Racing Algorithm for Configuring Metaheuristics , 2002, GECCO.

[24]  Eric Horvitz,et al.  Dynamic restart policies , 2002, AAAI/IAAI.

[25]  Holger H. Hoos,et al.  A mixture-model for the behaviour of SLS algorithms for SAT , 2002, AAAI/IAAI.

[26]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[27]  Yixin Diao,et al.  Generic Online Optimization of Multiple Configuration Parameters with Application to a Database Server , 2003, DSOM.

[28]  Alper Atamtürk,et al.  On the facets of the mixed–integer knapsack polyhedron , 2003, Math. Program..

[29]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[30]  Steven Minton,et al.  Automatically configuring constraint satisfaction programs: A case study , 1996, Constraints.

[31]  Holger H. Hoos,et al.  UBCSAT: An Implementation and Experimentation Environment for SLS Algorithms for SAT & MAX-SAT , 2004, SAT.

[32]  Mauro Birattari,et al.  The problem of tuning metaheuristics: as seen from the machine learning perspective , 2004 .

[33]  Charles Audet,et al.  Finding Optimal Algorithmic Parameters Using the Mesh Adaptive Direct Search Algorithm , 2004 .

[34]  Laurent Simon,et al.  Fifty-Five Solvers in Vancouver: The SAT 2004 Competition , 2004, SAT (Selected Papers.

[35]  Thomas Stützle,et al.  Stochastic Local Search: Foundations & Applications , 2004 .

[36]  Alper Atamtürk,et al.  A study of the lot-sizing polytope , 2004, Math. Program..

[37]  Emmanuel Zarpas,et al.  Benchmarking SAT Solvers for Bounded Model Checking , 2005, SAT.

[38]  Thomas Stützle,et al.  Efficient Stochastic Local Search for MPE Solving , 2005, IJCAI.

[39]  Eugene C. Freuder,et al.  Using CBR to Select Solution Strategies in Constraint Programming , 2005, ICCBR.

[40]  Mihai Oltean,et al.  Evolving Evolutionary Algorithms Using Linear Genetic Programming , 2005, Evolutionary Computation.

[41]  J. Christopher Beck,et al.  APPLYING MACHINE LEARNING TO LOW‐KNOWLEDGE CONTROL OF OPTIMIZATION ALGORITHMS , 2005, Comput. Intell..

[42]  Michael F. P. O'Boyle,et al.  Method-specific dynamic compilation using logistic regression , 2006, OOPSLA '06.

[43]  Thomas Bartz-Beielstein,et al.  Experimental Research in Evolutionary Computation - The New Experimentalism , 2010, Natural Computing Series.

[44]  D. Kudenko,et al.  Sequential Experiment Designs for Screening and Tuning Parameters of Stochastic Heuristics , 2006 .

[45]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms , 2006, CP.

[46]  Michael F. P. O'Boyle,et al.  Using machine learning to focus iterative optimization , 2006, International Symposium on Code Generation and Optimization (CGO'06).

[47]  CHARLES AUDET,et al.  Finding Optimal Algorithmic Parameters Using Derivative-Free Optimization , 2006, SIAM J. Optim..

[48]  Manuel Laguna,et al.  Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search , 2006, Oper. Res..

[49]  Jürgen Schmidhuber,et al.  Dynamic Algorithm Portfolios , 2006, AI&M.

[50]  Holger H. Hoos,et al.  A replica exchange Monte Carlo algorithm for protein folding in the HP model , 2007, BMC Bioinformatics.

[51]  Jürgen Schmidhuber,et al.  Learning Restart Strategies , 2007, IJCAI.

[52]  Alan J. Hu,et al.  Structural Abstraction of Software Verification Conditions , 2007, CAV.

[53]  Thomas Stützle,et al.  Improvement Strategies for the F-Race Algorithm: Sampling Design and Iterative Refinement , 2007, Hybrid Metaheuristics.

[54]  Kevin P. Murphy,et al.  Efficient parameter estimation for RNA secondary structure prediction , 2007, ISMB/ECCB.

[55]  Alan J. Hu,et al.  Boosting Verification by Automatic Tuning of Decision Procedures , 2007, Formal Methods in Computer Aided Design (FMCAD'07).

[56]  Thomas Stützle,et al.  Automatic Algorithm Configuration Based on Local Search , 2007, AAAI.

[57]  Ole J. Mengshoel,et al.  Understanding the role of noise in stochastic local search: Analysis and experiments , 2008, Artif. Intell..

[58]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[59]  H. Hoos.  Computer-Aided Design of High-Performance Algorithms , 2008 .

[60]  Mauro Brunato,et al.  Reactive Search and Intelligent Optimization , 2008 .

[61]  Frank Hutter,et al.  Automated configuration of algorithms for solving hard computational problems , 2009 .

[62]  Holger H. Hoos,et al.  An automatically configured modular algorithm for post enrollment course timetabling , 2009 .

[63]  Kevin Leyton-Brown,et al.  SATenstein: Automatically Building Local Search SAT Solvers from Components , 2009, IJCAI.

[64]  Sinan Gürel,et al.  A strong conic quadratic reformulation for machine-job assignment with controllable processing times , 2009, Oper. Res. Lett..

[65]  Kevin P. Murphy,et al.  An experimental investigation of model-based parameter optimisation: SPO and beyond , 2009, GECCO.

[66]  Kevin Leyton-Brown,et al.  Tradeoffs in the empirical evaluation of competing algorithm designs , 2010, Annals of Mathematics and Artificial Intelligence.

[67]  Sonja Kuhnt,et al.  Design and analysis of computer experiments , 2010 .

[68]  Athanasios K. Tsakalidis,et al.  Data Structures , 2011 .