LEAPSANDBOUNDS: A Method for Approximately Optimal Algorithm Configuration

We consider the problem of configuring general-purpose solvers to run efficiently on problem instances drawn from an unknown distribution. The goal of the configurator is to find a configuration that runs fast on average on most instances, and do so with the least amount of total work. It can run a chosen solver on a random instance until the solver finishes or a timeout is reached. We propose LeapsAndBounds, an algorithm that tests configurations on randomly selected problem instances for longer and longer time. We prove that the capped expected runtime of the configuration returned by LeapsAndBounds is close to the optimal expected runtime, while our algorithm's running time is near-optimal. Our results show that LeapsAndBounds is more efficient than the recent algorithm of Kleinberg et al. (2017), which, to our knowledge, is the only other algorithm configuration method with non-trivial theoretical guarantees. Experimental results on configuring a public SAT solver on a new benchmark dataset also stand witness to the superiority of our method.

[1]  David Zuckerman,et al.  Optimal Speedup of Las Vegas Algorithms , 1993, Inf. Process. Lett..

[2]  David Zuckerman,et al.  Optimal speedup of Las Vegas algorithms , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[3]  Klaus Adam.  Learning While Searching for the Best Alternative , 2001, J. Econ. Theory.

[4]  Thomas Stützle,et al.  A Racing Algorithm for Configuring Metaheuristics , 2002, GECCO.

[5]  Niklas Een,et al.  MiniSat v1.13 - A SAT Solver with Conflict-Clause Minimization , 2005 .

[6]  F. Hutter.  On the Potential of Automatic Algorithm Configuration , 2007 .

[7]  Csaba Szepesvári,et al.  Empirical Bernstein stopping , 2008, ICML '08.

[8]  Csaba Szepesvari,et al.  Efficient Stopping Rules , 2008 .

[9]  F. Hutter,et al.  ParamILS: An Automatic Algorithm Configuration Framework , 2009, J. Artif. Intell. Res..

[10]  Carlos Ansótegui,et al.  A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms , 2009, CP.

[11]  Dominik D. Freydenberger,et al.  Can We Learn to Gamble Efficiently? , 2010, COLT.

[12]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[13]  András György,et al.  Efficient Multi-Start Strategies for Local Search Algorithms , 2009, J. Artif. Intell. Res..

[14]  Kevin Leyton-Brown,et al.  Bayesian Optimization With Censored Response Data , 2013, ArXiv.

[15]  Yuri Malitsky,et al.  Model-Based Genetic Algorithms for Algorithm Configuration , 2015, IJCAI.

[16]  Leslie Pérez Cáceres,et al.  The irace package: Iterated racing for automatic algorithm configuration , 2016 .

[17]  Ameet Talwalkar,et al.  Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits , 2016, ArXiv.

[18]  Kevin Leyton-Brown,et al.  Efficiency Through Procrastination: Approximately Optimal Algorithm Configuration with Runtime Guarantees , 2017, IJCAI.