Quantifying Homogeneity of Instance Sets for Algorithm Configuration

Automated configuration procedures play an increasingly prominent role in realising the performance potential inherent in highly parametric solvers for a wide range of computationally challenging problems. However, these configuration procedures have difficulties when dealing with inhomogenous instance sets, where the relative difficulty of problem instances varies between configurations of the given parametric algorithm. In the literature, instance set homogeneity has been assessed using a qualitative, visual criterion based on heat maps. Here, we introduce two quantitative measures of homogeneity and empirically demonstrate these to be consistent with the earlier qualitative criterion. We also show that according to our measures, homogeneity increases when partitioning instance sets by means of clustering based on observed runtimes, and that the performance of a prominent automatic algorithm configurator increases on the resulting, more homogenous subsets.

[1]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms , 2006, CP.

[2]  David Lo,et al.  Instance-Based Parameter Tuning via Search Trajectory Similarity Clustering , 2011, LION.

[3]  Kevin Leyton-Brown,et al.  Tradeoffs in the empirical evaluation of competing algorithm designs , 2010, Annals of Mathematics and Artificial Intelligence.

[4]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[5]  Kevin Leyton-Brown,et al.  Hierarchical Hardness Models for SAT , 2007, CP.

[6]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[7]  Alfonso Gerevini,et al.  Generating Fast Domain-Specific Planners by Automatically Configuring a Generic Parameterised Planner , 2011 .

[8]  Holger H. Hoos,et al.  Programming by optimization , 2012, Commun. ACM.

[9]  Thomas Stützle,et al.  A Racing Algorithm for Configuring Metaheuristics , 2002, GECCO.

[10]  Emmanuel Zarpas,et al.  Benchmarking SAT Solvers for Bounded Model Checking , 2005, SAT.

[11]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[12]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[13]  Karem A. Sakallah,et al.  Theory and Applications of Satisfiability Testing - SAT 2011 - 14th International Conference, SAT 2011, Ann Arbor, MI, USA, June 19-22, 2011. Proceedings , 2011, SAT.

[14]  Christian Bessière.  Principles and Practice of Constraint Programming - CP 2007, 13th International Conference, CP 2007, Providence, RI, USA, September 23-27, 2007, Proceedings , 2007, CP.

[15]  Roberto Rossi,et al.  Synthesizing Filtering Algorithms for Global Chance-Constraints , 2009, CP.

[16]  Thomas Stützle,et al.  Automatic Algorithm Configuration Based on Local Search , 2007, AAAI.

[17]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[18]  Alan J. Hu,et al.  Boosting Verification by Automatic Tuning of Decision Procedures , 2007 .

[19]  Holger H. Hoos,et al.  Captain Jack: New Variable Selection Heuristics in Local Search for SAT , 2011, SAT.

[20]  Laurence A. Wolsey,et al.  Integration of AI and OR Techniques in Constraint Programming for Combinatorial Optimization Problems, 4th International Conference, CPAIOR 2007, Brussels, Belgium, May 23-26, 2007, Proceedings , 2007, CPAIOR.

[21]  Yuri Malitsky,et al.  ISAC - Instance-Specific Algorithm Configuration , 2010, ECAI.

[22]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[23]  Stephen F. Smith,et al.  Combining Multiple Heuristics Online , 2007, AAAI.

[24]  Marius Thomas Lindauer,et al.  A Portfolio Solver for Answer Set Programming: Preliminary Report , 2011, LPNMR.

[25]  Greg Hamerly,et al.  Learning the k in k-means , 2003, NIPS.

[26]  J. Neyman,et al.  Mathematical Statistics and Probability , 1962 .

[27]  Michela Milano,et al.  Learning Techniques for Automatic Algorithm Portfolio Selection , 2004, ECAI.

[28]  Kousha Etessami,et al.  Recursive Markov chains, stochastic grammars, and monotone systems of nonlinear equations , 2005, JACM.

[29]  Felix Naumann,et al.  Data fusion , 2009, CSUR.

[30]  Ashiqur R. KhudaBukhsh,et al.  SATenstein: automatically building local search SAT solvers from components , 2009, IJCAI 2009.

[31]  Lars Kotthoff,et al.  A Preliminary Evaluation of Machine Learning in Algorithm Selection for Search Problems , 2011, SOCS.

[32]  Martin Gebser,et al.  Conflict-Driven Answer Set Solving , 2007, IJCAI.

[33]  Wolfgang Faber,et al.  Logic Programming and Nonmonotonic Reasoning , 2011, Lecture Notes in Computer Science.

[34]  Kevin Leyton-Brown,et al.  Automated Configuration of Mixed Integer Programming Solvers , 2010, CPAIOR.

[35]  Yoav Shoham,et al.  Empirical hardness models: Methodology and a case study on combinatorial auctions , 2009, JACM.

[36]  Yuri Malitsky,et al.  Algorithm Selection and Scheduling , 2011, CP.

[37]  Lakhdar Sais,et al.  ManySAT: a Parallel SAT Solver , 2009, J. Satisf. Boolean Model. Comput..

[38]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[39]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[40]  Kevin Leyton-Brown,et al.  Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection , 2010, AAAI.

[41]  H. Hoos.  Programming by Optimisation , 2010 .

[42]  Alex S. Fukunaga,et al.  Variable-Selection Heuristics in Local Search for SAT , 1997, AAAI/IAAI.

[43]  Alex M. Andrew,et al.  Knowledge Representation, Reasoning and Declarative Problem Solving , 2004 .

[44]  Carlos Ansótegui,et al.  A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms , 2009, CP.

[45]  J. H. Ward.  Hierarchical Grouping to Optimize an Objective Function , 1963 .

[46]  Frédéric Benhamou.  Principles and Practice of Constraint Programming - CP 2006, 12th International Conference, CP 2006, Nantes, France, September 25-29, 2006, Proceedings , 2006, CP.