Algorithm Selection for Combinatorial Search Problems: A survey

The Algorithm Selection Problem is concerned with selecting the best algorithm to solve a given problem on a case-by-case basis. It has become especially relevant in the last decade, as researchers are increasingly investigating how to identify the most suitable existing algorithm for solving a problem instead of developing new algorithms. This survey presents an overview of this work focusing on the contributions made in the area of combinatorial search problems, where Algorithm Selection techniques have achieved signicant performance improvements. We unify and organise the vast literature according to criteria that determine Algorithm Selection systems in practice. The comprehensive classication of approaches identies and analyses the dierent directions from which Algorithm Selection has been approached. This paper contrasts and compares dierent methods for solving the problem as well as ways of using these solutions. It closes by identifying directions of current and future research.

[1]  Diane J. Cook,et al.  Maximizing the Benefits of Parallel Search Using Machine Learning , 1997, AAAI/IAAI.

[2]  David Maxwell Chickering,et al.  A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report) , 2001, Electron. Notes Discret. Math..

[3]  Yoav Shoham,et al.  Empirical hardness models: Methodology and a case study on combinatorial auctions , 2009, JACM.

[4]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[5]  Jürgen Schmidhuber,et al.  A Neural Network Model for Inter-problem Adaptive Online Time Allocation , 2005, ICANN.

[6]  Matteo Gagliolo,et al.  Online Dynamic Algorithm Portfolios: Minimizing the computational cost of problem solving, Informatics , 2010 .

[7]  Mohamed S. Kamel,et al.  ODEXPERT: an expert system to select numerical solvers for initial value ODE systems , 1993, TOMS.

[8]  Luca Pulina,et al.  Collaborative Expert Portfolio Management , 2010, AAAI.

[9]  Erez Karpas,et al.  To Max or Not to Max: Online Learning for Speeding Up Optimal Planning , 2010, AAAI.

[10]  María Cristina Riff,et al.  DVRP: a hard dynamic combinatorial optimisation problem tackled by an evolutionary hyper-heuristic , 2010, J. Heuristics.

[11]  J. Christopher Beck,et al.  Low-Knowledge Algorithm Control , 2004, AAAI.

[12]  Jonathan. Sillito.  Improvements to and estimating the cost of backtracking algorithms for constraint satisfaction problems , 2000 .

[13]  Stephen F. Smith,et al.  Combining Multiple Heuristics Online , 2007, AAAI.

[14]  Lars Kotthoff,et al.  An evaluation of machine learning in algorithm selection for search problems , 2012, AI Commun..

[15]  Peter C. Cheeseman,et al.  Where the Really Hard Problems Are , 1991, IJCAI.

[16]  Paul E. Utgoff,et al.  Perceptron Trees : A Case Study in ybrid Concept epresentations , 1999 .

[17]  Michail G. Lagoudakis,et al.  Learning to Select Branching Rules in the DPLL Procedure for Satisfiability , 2001, Electron. Notes Discret. Math..

[18]  Marek Petrik,et al.  Learning parallel portfolios of algorithms , 2006, Annals of Mathematics and Artificial Intelligence.

[19]  Sanjukta Bhowmick,et al.  Towards Low-Cost, High-Accuracy Classifiers for Linear Solver Selection , 2009, ICCS.

[20]  Marie desJardins,et al.  What Makes Planners Predictable? , 2008, ICAPS.

[21]  William H. Hsu,et al.  A Learning-Based Algorithm Selection Meta-reasoner for the Real-Time MPE Problem , 2004, Australian Conference on Artificial Intelligence.

[22]  Alan Edelman,et al.  PetaBricks: a language and compiler for algorithmic choice , 2009, PLDI '09.

[23]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[24]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[25]  Michela Milano,et al.  Learning Techniques for Automatic Algorithm Portfolio Selection , 2004, ECAI.

[26]  Lars Kotthoff,et al.  Machine learning for constraint solver design -- A case study for the alldifferent constraint , 2010, ArXiv.

[27]  W. Hsu,et al.  Algorithm selection for sorting and probabilistic inference: a machine learning-based approach , 2003 .

[28]  Dorothy E. Setliff,et al.  Knowledge-based constraint-driven software synthesis , 1992, Proceedings of the Seventh Knowledge-Based Software Engineering Conference.

[29]  Stephen F. Smith,et al.  Restart Schedules for Ensembles of Problem Instances , 2007, AAAI.

[30]  Jun Wang,et al.  Optimizing time warp simulation with reinforcement learning techniques , 2007, 2007 Winter Simulation Conference.

[31]  Adele E. Howe,et al.  Exploiting Competitive Planner Performance , 1999, ECP.

[32]  Eamonn Cahill,et al.  Knowledge-based algorithm construction for real-world engineering PDEs , 1994 .

[33]  Jürgen Schmidhuber,et al.  Adaptive Online Time Allocation to Search Algorithms , 2004, ECML.

[34]  W. Armstrong,et al.  Dynamic Algorithm Selection Using Reinforcement Learning , 2006, 2006 International Workshop on Integrating AI and Data Mining.

[35]  Michail G. Lagoudakis,et al.  Algorithm Selection using Reinforcement Learning , 2000, ICML.

[36]  Toby Walsh,et al.  Restart Strategy Selection Using Machine Learning Techniques , 2009, SAT.

[37]  David Tolpin,et al.  Rational Deployment of CSP Heuristics , 2011, IJCAI.

[38]  Roland Ewald,et al.  Automatic Algorithm Selection for Complex Simulation Problems , 2011, Vieweg+Teubner Verlag.

[39]  David W. Aha,et al.  Generalizing from Case studies: A Case Study , 1992, ML.

[40]  Victor Eijkhout,et al.  Self-Adapting Linear Algebra Algorithms and Software , 2005, Proceedings of the IEEE.

[41]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[42]  Pat Langley,et al.  Learning Search Strategies through Discrimination , 1983, Int. J. Man Mach. Stud..

[43]  Yishay Mansour,et al.  Combining Multiple Heuristics , 2006, STACS.

[44]  Predrag Janicic,et al.  Instance-Based Selection of Policies for SAT Solvers , 2009, SAT.

[45]  Jürgen Schmidhuber,et al.  Learning dynamic algorithm portfolios , 2006, Annals of Mathematics and Artificial Intelligence.

[46]  Efstathios Stamatatos,et al.  Learning How to Propagate Using Random Probing , 2009, CPAIOR.

[47]  Kevin Leyton-Brown,et al.  Parallel Algorithm Configuration , 2012, LION.

[48]  Tzu-Yi Chen,et al.  On Using Reinforcement Learning to Solve Sparse Linear Systems , 2008, ICCS.

[49]  Eugene C. Freuder,et al.  Using CBR to Select Solution Strategies in Constraint Programming , 2005, ICCBR.

[50]  Oren Etzioni,et al.  PRODIGY: an integrated architecture for planning and learning , 1991, SGAR.

[51]  Patricia Diane Hough,et al.  Modern Machine Learning for Automatic Optimization Algorithm Selection. , 2006 .

[52]  Gerald DeJong,et al.  COMPOSER: A Probabilistic Solution to the Utility Problem in Speed-Up Learning , 1992, AAAI.

[53]  Jürgen Schmidhuber,et al.  Towards Distributed Algorithm Portfolios , 2008, DCAI.

[54]  Steven Minton,et al.  Integrating Heuristics for Constraint Satisfaction Problems: A Case Study , 1993, AAAI.

[55]  Susan L. Epstein,et al.  The Adaptive Constraint Engine , 2002, CP.

[56]  Michel Lemaître,et al.  Branch and Bound Algorithm Selection by Performance Prediction , 1998, AAAI/IAAI.

[57]  Pedro M. Domingos,et al.  How to Get a Free Lunch: A Simple Cost Model for Machine Learning Applications , 1998 .

[58]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms , 2006, CP.

[59]  Kevin Leyton-Brown,et al.  Evaluating Component Solver Contributions to Portfolio-Based Algorithm Selectors , 2012, SAT.

[60]  Stephen F. Smith,et al.  New Techniques for Algorithm Portfolio Design , 2008, UAI.

[61]  Grigorios Tsoumakas,et al.  Learning Rules for Adaptive Planning , 2003, ICAPS.

[62]  Kostas Stergiou.  Heuristics for dynamically adapting propagation in constraint satisfaction problems , 2009, AI Commun..

[63]  Tad Hogg,et al.  Phase Transitions and the Search Problem , 1996, Artif. Intell..

[64]  Marin Bougeret,et al.  Combining multiple heuristics on discrete resources , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[65]  Eugene Fink.  Statistical Selection Among Problem-Solving Methods. , 1997 .

[66]  Marius Thomas Lindauer,et al.  A Portfolio Solver for Answer Set Programming: Preliminary Report , 2011, LPNMR.

[67]  Eugene Fink,et al.  How to Solve It Automatically: Selection Among Problem Solving Methods , 1998, AIPS.

[68]  Lu Hong,et al.  Groups of diverse problem solvers can outperform groups of high-ability problem solvers. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[69]  J. Christopher Beck,et al.  APPLYING MACHINE LEARNING TO LOW‐KNOWLEDGE CONTROL OF OPTIMIZATION ALGORITHMS , 2005, Comput. Intell..

[70]  Brahim Hnich,et al.  Making Choices Using Structure at the Instance Level within a Case Based Reasoning Framework , 2004, CPAIOR.

[71]  Carlos Ansótegui,et al.  A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms , 2009, CP.

[72]  Stephen F. Smith,et al.  The Max K-Armed Bandit: A New Model of Exploration Applied to Search Heuristic Selection , 2005, AAAI.

[73]  Carlos Soares,et al.  A Meta-Learning Method to Select the Kernel Width in Support Vector Regression , 2004, Machine Learning.

[74]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[75]  Mark Wallace,et al.  An Instance of Adaptive Constraint Propagation , 1996, CP.

[76]  Yoav Shoham,et al.  Understanding Random SAT: Beyond the Clauses-to-Variables Ratio , 2004, CP.

[77]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[78]  Eugene C. Freuder,et al.  Using Case-Based Reasoning to Write Constraint Programs , 2003, CP.

[79]  Lars Kotthoff,et al.  Learning When to Use Lazy Learning in Constraint Solving , 2010, ECAI.

[80]  Eric A. Brewer,et al.  High-level optimization via automated statistical modeling , 1995, PPOPP '95.

[81]  Naren Ramakrishnan,et al.  How to get a Free Lunch (at no cost) , 1999 .

[82]  John R. Rice,et al.  PYTHIA: a knowledge-based system to select scientific algorithms , 1996, TOMS.

[83]  Yoav Shoham,et al.  Learning the Empirical Hardness of Optimization Problems: The Case of Combinatorial Auctions , 2002, CP.

[84]  Kevin Leyton-Brown,et al.  Hierarchical Hardness Models for SAT , 2007, CP.

[85]  Ryan Williams,et al.  Confronting hardness using a hybrid approach , 2006, SODA '06.

[86]  Alexander Nareyek,et al.  Choosing search heuristics by non-stationary reinforcement learning , 2004 .

[87]  Hilan Bensusan,et al.  Meta-Learning by Landmarking Various Learning Algorithms , 2000, ICML.

[88]  Yuri Malitsky,et al.  Algorithm Selection and Scheduling , 2011, CP.

[89]  Steven Minton,et al.  Automatically configuring constraint satisfaction programs: A case study , 1996, Constraints.

[90]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[91]  Stephen F. Smith,et al.  Using online algorithms to solve np-hard problems more efficiently in practice , 2007 .

[92]  Horst Samulowitz,et al.  Learning to Solve QBF , 2007, AAAI.

[93]  Carla E. Brodley,et al.  Addressing the Selective Superiority Problem: Automatic Algorithm/Model Class Selection , 1993 .

[94]  Kevin Leyton-Brown,et al.  Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection , 2010, AAAI.

[95]  Carlos Soares,et al.  A Comparison of Ranking Methods for Classification Algorithm Selection , 2000, ECML.

[96]  Yuri Malitsky,et al.  Instance-Specic Algorithm Conguration , 2012 .

[97]  Michèle Sebag,et al.  Online Heuristic Selection in Constraint Programming , 2009 .

[98]  Yuri Malitsky,et al.  Non-Model-Based Algorithm Portfolios for SAT , 2011, SAT.

[99]  H. Terashima-Marín,et al.  Evolution of Constraint Satisfaction strategies in examination timetabling , 1999 .

[100]  Eric Bauer,et al.  An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants , 1999, Machine Learning.

[101]  Joaquin Vanschoren,et al.  Using active testing and meta-level information for selection of classification algorithms , 2010 .

[102]  François Laburthe,et al.  A Meta-Heuristic Factory for Vehicle Routing Problems , 1999, CP.

[103]  Sanja Petrovic,et al.  Case-Based Reasoning as a Heuristic Selector in a Hyper-Heuristic for Course Timetabling Problems , 2002 .

[104]  Kevin Leyton-Brown,et al.  SATzilla2009: an Automatic Algorithm Portfolio for SAT , 2008 .

[105]  Harry Zhang,et al.  Switching among Non-Weighting, Clause Weighting, and Variable Weighting in Local Search for SAT , 2008, CP.

[106]  Alvin C. M. Kwan,et al.  An Attempt to Map the Performance of a Range of Algorithm and Heuristic Combinations , 2007 .

[107]  Susan L. Epstein,et al.  Learning Algorithm Portfolios for Parallel Execution , 2012, LION.

[108]  Risto Miikkulainen,et al.  Latent class models for algorithm portfolio methods , 2010, AAAI 2010.

[109]  H. Yu,et al.  An adaptive algorithm selection framework , 2004, Proceedings. 13th International Conference on Parallel Architecture and Compilation Techniques, 2004. PACT 2004..

[110]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[111]  Jürgen Schmidhuber,et al.  Impact of Censored Sampling on the Performance of Restart Strategies , 2006, CP.

[112]  Kevin Leyton-Brown,et al.  : The Design and Analysis of an Algorithm Portfolio for SAT , 2007, CP.

[113]  Bart Selman,et al.  Algorithm Portfolio Design: Theory vs. Practice , 1997, UAI.

[114]  Susan L. Epstein,et al.  Collaborative Learning for Constraint Solving , 2001, CP.

[115]  J. Christopher Beck,et al.  Simple Rules for Low-Knowledge Algorithm Selection , 2004, CPAIOR.

[116]  W. Spears,et al.  For Every Generalization Action, Is There Really an Equal and Opposite Reaction? , 1995, ICML.

[117]  Alex S. Fukunaga,et al.  Automated discovery of composite SAT variable-selection heuristics , 2002, AAAI/IAAI.

[118]  Samir A. Mohamed Elsayed,et al.  Synthesis of Search Algorithms from High-Level CP Models , 2011, CP.

[119]  Jürgen Schmidhuber,et al.  Algorithm portfolio selection as a bandit problem with unbounded losses , 2011, Annals of Mathematics and Artificial Intelligence.

[120]  L. D. Whitley,et al.  Empirical modeling and analysis of local search algorithms for the job-shop scheduling problem , 2003 .

[121]  Steven Minton,et al.  Selecting the Right Heuristic Algorithm: Runtime Performance Predictors , 1996, Canadian Conference on AI.

[122]  Adelinde M. Uhrmacher,et al.  Selecting Simulation Algorithm Portfolios by Genetic Algorithms , 2010, 2010 IEEE Workshop on Principles of Advanced and Distributed Simulation.

[123]  Edward P. K. Tsang,et al.  A Context for Constraint Satisfaction Problem Formulation Selection , 2001, Constraints.

[124]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[125]  Holger H. Hoos,et al.  Programming by optimization , 2012, Commun. ACM.

[126]  P. van Beek,et al.  On Portfolios for Backtracking Search in the Presence of Deadlines , 2007 .

[127]  Tad Hogg,et al.  An Economics Approach to Hard Computational Problems , 1997, Science.

[128]  Luca Pulina,et al.  A self-adaptive multi-engine solver for quantified Boolean formulas , 2009, Constraints.

[129]  Marek Petrik,et al.  Statistically Optimal Combination of Algorithms , 2004 .

[130]  Alex Fukunaga,et al.  Genetic algorithm portfolios , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[131]  D. Wolpert.  The Supervised Learning No-Free-Lunch Theorems , 2002 .

[132]  J. Christopher Beck,et al.  Dynamic problem structure analysis as a basis for constraint-directed scheduling heuristics , 2000, Artif. Intell..

[133]  Alfonso Gerevini,et al.  An Automatically Configurable Portfolio-based Planner with Macro-actions: PbP , 2009, ICAPS.

[134]  Lawrence Rauchwerger,et al.  An Adaptive Algorithm Selection Framework for Reduction Parallelization , 2006, IEEE Transactions on Parallel and Distributed Systems.