An evaluation of machine learning in algorithm selection for search problems

Machine learning is an established method of selecting algorithms to solve hard search problems. Despite this, to date no systematic comparison and evaluation of the different techniques has been performed and the performance of existing systems has not been critically compared with other approaches. We compare the performance of a large number of different machine learning techniques from different machine learning methodologies on five data sets of hard algorithm selection problems from the literature. In addition to well-established approaches, for the first time we also apply statistical relational learning to this problem. We demonstrate that there is significant scope for improvement both compared with existing systems and in general. To guide practitioners, we close by giving clear recommendations as to which machine learning techniques are likely to achieve good performance in the context of algorithm selection problems. In particular, we show that linear regression and alternating decision trees have a very high probability of achieving better performance than always selecting the single best algorithm.

[1]  W. Spears,et al.  For Every Generalization Action, Is There Really an Equal and Opposite Reaction? , 1995, ICML.

[2]  Christopher K. Riesbeck,et al.  Inside Case-Based Reasoning , 1989 .

[3]  John R. Rice,et al.  PYTHIA: a knowledge-based system to select scientific algorithms , 1996, TOMS.

[4]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[5]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[6]  Luca Pulina,et al.  A Multi-engine Solver for Quantified Boolean Formulas , 2007, CP.

[7]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[8]  A. Howe,et al.  Learned Models of Performance for Many Planners , 2007 .

[9]  Stephen F. Smith,et al.  New Techniques for Algorithm Portfolio Design , 2008, UAI.

[10]  Risto Miikkulainen,et al.  Latent class models for algorithm portfolio methods , 2010, AAAI 2010.

[11]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[12]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[13]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[14]  Lars Kotthoff,et al.  Machine learning for constraint solver design -- A case study for the alldifferent constraint , 2010, ArXiv.

[15]  Yuri Malitsky,et al.  ISAC - Instance-Specific Algorithm Configuration , 2010, ECAI.

[16]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[17]  Ivana Kruijff-Korbayová,et al.  A Portfolio Approach to Algorithm Selection , 2003, IJCAI.

[18]  Steven Minton,et al.  Automatically configuring constraint satisfaction programs: A case study , 1996, Constraints.

[19]  Kevin Leyton-Brown,et al.  Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection , 2010, AAAI.

[20]  Robert C. Holte,et al.  Very Simple Classification Rules Perform Well on Most Commonly Used Datasets , 1993, Machine Learning.

[21]  V. Eijkhout,et al.  Application of Machine Learning in Selecting Sparse Linear Solvers ⋆ , 2006 .

[22]  Diane J. Cook,et al.  Maximizing the Benefits of Parallel Search Using Machine Learning , 1997, AAAI/IAAI.

[23]  Lars Kotthoff,et al.  Ensemble Classification for Constraint Solver Configuration , 2010, CP.

[24]  Eugene Fink,et al.  How to Solve It Automatically: Selection Among Problem Solving Methods , 1998, AIPS.

[25]  Alfonso Gerevini,et al.  An Automatically Configurable Portfolio-based Planner with Macro-actions: PbP , 2009, ICAPS.

[26]  Michail G. Lagoudakis,et al.  Algorithm Selection using Reinforcement Learning , 2000, ICML.

[27]  D. Wolpert.  The Supervised Learning No-Free-Lunch Theorems , 2002 .

[28]  Eoin O'Mahony,et al.  Using Case-based Reasoning in an Algorithm Portfolio for Constraint Solving ? , 2008 .

[29]  R. Geoff Dromey,et al.  An algorithm for the selection problem , 1986, Softw. Pract. Exp..

[30]  Ben Taskar,et al.  Introduction to Statistical Relational Learning (Adaptive Computation and Machine Learning) , 2007 .

[31]  Michela Milano,et al.  Learning Techniques for Automatic Algorithm Portfolio Selection , 2004, ECAI.

[32]  Thorsten Joachims,et al.  Training linear SVMs in linear time , 2006, KDD '06.

[33]  Eugene C. Freuder,et al.  Using CBR to Select Solution Strategies in Constraint Programming , 2005, ICCBR.

[34]  Michel Lemaître,et al.  Branch and Bound Algorithm Selection by Performance Prediction , 1998, AAAI/IAAI.

[35]  Oren Etzioni,et al.  PRODIGY: an integrated architecture for planning and learning , 1991, SGAR.

[36]  Patricia Diane Hough,et al.  Modern Machine Learning for Automatic Optimization Algorithm Selection. , 2006 .

[37]  Lars Kotthoff,et al.  Learning When to Use Lazy Learning in Constraint Solving , 2010, ECAI.

[38]  Luca Pulina,et al.  A self-adaptive multi-engine solver for quantified Boolean formulas , 2009, Constraints.

[39]  Yoav Shoham,et al.  A portfolio approach to algorithm select , 2003, IJCAI 2003.

[40]  Luca Pulina,et al.  Collaborative Expert Portfolio Management , 2010, AAAI.

[41]  Edward P. K. Tsang,et al.  Adaptive Constraint Satisfaction: The Quickest First Principle , 1996, ECAI.

[42]  William H. Hsu,et al.  A Learning-Based Algorithm Selection Meta-reasoner for the Real-Time MPE Problem , 2004, Australian Conference on Artificial Intelligence.