The Algorithm Selection Competition Series 2015-17

The algorithm selection problem is to choose the most suitable algorithm for solving a given problem instance and thus, it leverages the complementarity between different approaches that is present in many areas of AI. We report on the state of the art in algorithm selection, as defined by the Algorithm Selection Competition series 2015 to 2017. The results of these competitions show how the state of the art improved over the years. Although performance in some cases is very promising, there is still room for improvement in other cases. Finally, we provide insights into why some scenarios are hard, and pose challenges to the community on how to advance the current state of the art.

[1]  Marius Thomas Lindauer,et al.  AutoFolio: An Automatically Configured Algorithm Selector , 2015, J. Artif. Intell. Res..

[2]  Michèle Sebag,et al.  ASAP.V2 and ASAP.V3: Sequential optimization of an Algorithm Selector and a Scheduler , 2017, OASC.

[3]  Brandon M. Malone,et al.  AS-ASL: Algorithm Selection with Auto-sklearn , 2017, OASC.

[4]  Barry O'Sullivan,et al.  The ICON Challenge on Algorithm Selection , 2015, AI Mag..

[5]  David Maxwell Chickering,et al.  A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report) , 2001, Electron. Notes Discret. Math..

[6]  Joaquin Vanschoren,et al.  Fast Algorithm Selection Using Learning Curves , 2015, IDA.

[7]  Kevin Leyton-Brown,et al.  Evaluating Component Solver Contributions to Portfolio-Based Algorithm Selectors , 2012, SAT.

[8]  Heike Trautmann,et al.  Improving the State of the Art in Inexact TSP Solving Using Per-Instance Algorithm Selection , 2015, LION.

[9]  Kevin Leyton-Brown,et al.  Bias in Algorithm Portfolio Performance Evaluation , 2016, IJCAI.

[10]  F. Hutter,et al.  Hydra-MIP : Automated Algorithm Configuration and Selection for Mixed Integer Programming , 2011 .

[11]  Lars Kotthoff,et al.  Open Algorithm Selection Challenge 2017: Setup and Scenarios , 2017, OASC.

[12]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[13]  Torsten Schaub,et al.  Automatic construction of parallel portfolios via algorithm configuration , 2017, Artif. Intell..

[14]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[15]  Maurizio Gabbrielli,et al.  SUNNY: a Lazy Portfolio Approach for Constraint Solving , 2014, Theory Pract. Log. Program..

[16]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[17]  Lars Kotthoff,et al.  LLAMA: Leveraging Learning to Automatically Manage Algorithms , 2013, ArXiv.

[18]  Peter J. Stuckey,et al.  The MiniZinc Challenge 2008-2013 , 2014, AI Mag..

[19]  R. Geoff Dromey,et al.  An algorithm for the selection problem , 1986, Softw. Pract. Exp..

[20]  Jan N. van Rijn,et al.  Hyperparameter Importance Across Datasets , 2017, KDD.

[21]  Barry O'Sullivan,et al.  Proteus: A Hierarchical Portfolio of Solvers and Transformations , 2013, CPAIOR.

[22]  Meinolf Sellmann,et al.  Snappy: A Simple Algorithm Portfolio , 2013, SAT.

[23]  Carlos Ansótegui,et al.  A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms , 2009, CP.

[24]  Brandon M. Malone,et al.  Empirical hardness of finding optimal Bayesian network structures: algorithm selection and runtime prediction , 2017, Machine Learning.

[25]  J. Carpenter.  May the best analyst win. , 2011, Science.

[26]  Geoff Holmes,et al.  The online performance estimation framework: heterogeneous ensemble learning for data streams , 2017, Machine Learning.

[27]  Bernd Bischl,et al.  ASlib: A benchmark library for algorithm selection , 2015, Artif. Intell..

[28]  Yuri Malitsky,et al.  MaxSAT by Improved Instance-Specific Algorithm Configuration , 2014, AAAI.

[29]  Kevin Leyton-Brown,et al.  OASC-2017: *Zilla Submission , 2017, OASC.

[30]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[31]  Luís Torgo,et al.  OpenML: networked science in machine learning , 2014, SKDD.

[32]  Jayant R. Haritsa,et al.  Plan Bouquets , 2016, ACM Trans. Database Syst..

[33]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[34]  Frank Hutter,et al.  Initializing Bayesian Hyperparameter Optimization via Meta-Learning , 2015, AAAI.

[35]  F. Post,et al.  An Economics Approach to Hard Computational Problems , 1997 .

[36]  van,et al.  Massively collaborative machine learning , 2016 .

[37]  Marius Thomas Lindauer,et al.  claspfolio 2: Advances in Algorithm Selection for Answer Set Programming , 2014, Theory and Practice of Logic Programming.

[38]  Luca Pulina,et al.  A multi-engine approach to answer-set programming* , 2013, Theory and Practice of Logic Programming.

[39]  Franz Wotawa,et al.  Improving Abductive Diagnosis Through Structural Features: A Meta-Approach , 2016, DARe@ECAI.

[40]  M. Lindauer,et al.  Surviving Solver Sensitivity: An ASP Practitioner's Guide , 2012, ICLP.

[41]  Lars Kotthoff,et al.  Algorithm Selection for Combinatorial Search Problems: A Survey , 2012, AI Mag..

[42]  Lars Kotthoff,et al.  Hybrid Regression-Classification Models for Algorithm Selection , 2012, ECAI.

[43]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[44]  Michèle Sebag,et al.  Algorithm Selector and Prescheduler in the ICON Challenge , 2018, Bioinspired Heuristics for Optimization.

[45]  Stefan Szeider,et al.  Portfolio-Based Algorithm Selection for Circuit QBFs , 2018, CP.

[46]  Lars Kottho,et al.  Algorithm Selection for Combinatorial Search Problems: A survey , 2012 .

[47]  Kevin Leyton-Brown,et al.  Algorithm Runtime Prediction: Methods and Evaluation (Extended Abstract) , 2015, IJCAI.

[48]  Yuri Malitsky,et al.  Algorithm Selection and Scheduling , 2011, CP.

[49]  t. liu.  SUNNY with Algorithm Configuration , 2017 .

[50]  Bart Selman,et al.  Algorithm portfolios , 2001, Artif. Intell..

[51]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[52]  Simona Perri,et al.  I-DLV+MS: Preliminary Report on an Automatic ASP Solver Selector , 2017, RCRA@AI*IA.

[53]  Marius Thomas Lindauer,et al.  aspeed: Solver scheduling via answer set programming 1 , 2014, Theory and Practice of Logic Programming.