Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms

Machine learning can be used to build models that predict the run-time of search algorithms for hard combinatorial problems. Such empirical hardness models have previously been studied for complete, deterministic search algorithms. In this work, we demonstrate that such models can also make surprisingly accurate predictions of the run-time distributions of incomplete and randomized search methods, such as stochastic local search algorithms. We also show for the first time how information about an algorithm's parameter settings can be incorporated into a model, and how such models can be used to automatically adjust the algorithm's parameters on a per-instance basis in order to optimize its performance. Empirical results for Novelty+ and SAPS on structured and unstructured SAT instances show very good predictive performance and significant speedups of our automatically determined parameter settings when compared to the default and best fixed distribution-specific parameter settings.

[1]  Steven Minton,et al.  Automatically configuring constraint satisfaction programs: A case study , 1996, Constraints.

[2]  Yoav Shoham,et al.  Understanding Random SAT: Beyond the Clauses-to-Variables Ratio , 2004, CP.

[3]  Holger H. Hoos,et al.  Scaling and Probabilistic Smoothing: Efficient Dynamic Local Search for SAT , 2002, CP.

[4]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[5]  Matthias F. Stallmann,et al.  On SAT instance classes and a method for reliable performance experiments with SAT solvers , 2005, Annals of Mathematics and Artificial Intelligence.

[6]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[7]  Thomas Stützle,et al.  A Racing Algorithm for Configuring Metaheuristics , 2002, GECCO.

[8]  John Thornton,et al.  Clause Weighting Local Search for SAT , 2005, Journal of Automated Reasoning.

[9]  Bart Selman,et al.  Heavy-Tailed Phenomena in Satisfiability and Constraint Satisfaction Problems , 2000, Journal of Automated Reasoning.

[10]  Frank Hutter,et al.  Parameter Adjustment Based on Performance Prediction: Towards an Instance-Aware Problem Solver , 2005 .

[11]  Bart Selman,et al.  Problem Structure in the Presence of Perturbations , 1997, AAAI/IAAI.

[12]  Michail G. Lagoudakis,et al.  Learning to Select Branching Rules in the DPLL Procedure for Satisfiability , 2001, Electron. Notes Discret. Math..

[13]  Toby Walsh,et al.  Morphing: Combining Structure and Randomness , 1999, AAAI/IAAI.

[14]  J. Christopher Beck,et al.  APPLYING MACHINE LEARNING TO LOW‐KNOWLEDGE CONTROL OF OPTIMIZATION ALGORITHMS , 2005, Comput. Intell..

[15]  David Maxwell Chickering,et al.  A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report) , 2001, Electron. Notes Discret. Math..

[16]  Edward P. K. Tsang,et al.  Adaptive Constraint Satisfaction: The Quickest First Principle , 1996, ECAI.

[17]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[18]  Teofilo F. Gonzalez,et al.  Reactive Search: Machine Learning for Memory-Based Heuristics , 2007 .

[19]  Holger H. Hoos,et al.  On the Run-time Behaviour of Stochastic Local Search Algorithms for SAT , 1999, AAAI/IAAI.

[20]  Manuel Laguna,et al.  Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search , 2006, Oper. Res..

[21]  Thomas Stützle,et al.  Stochastic Local Search: Foundations & Applications , 2004 .

[22]  Ron Kohavi,et al.  Automatic Parameter Selection by Minimizing Estimated Error , 1995, ICML.

[23]  Bart Selman,et al.  Evidence for Invariants in Local Search , 1997, AAAI/IAAI.

[24]  Eugene C. Freuder,et al.  Using CBR to Select Solution Strategies in Constraint Programming , 2005, ICCBR.

[25]  Susan L. Epstein,et al.  The Adaptive Constraint Engine , 2002, CP.

[26]  Michel Lemaître,et al.  Branch and Bound Algorithm Selection by Performance Prediction , 1998, AAAI/IAAI.

[27]  Biplav Srivastava,et al.  Domain-Dependent Parameter Selection of Search-based Algorithms Compatible with User Performance Criteria , 2005, AAAI.

[28]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[29]  Holger H. Hoos,et al.  An adaptive noise mechanism for walkSAT , 2002, AAAI/IAAI.

[30]  Henry A. Kautz,et al.  Auto-Walksat: A Self-Tuning Implementation of Walksat , 2001, Electron. Notes Discret. Math..