Optimisation and Generalisation: Footprints in Instance Space

The chief purpose of research in optimisation is to understand how to design (or choose) the most suitable algorithm for a given distribution of problem instances. Ideally, when an algorithm is developed for specific problems, the boundaries of its performance should be clear, and we expect estimates of reasonably good performance within and (at least modestly) outside its 'seen' instance distribution. However, we show that these ideals are highly over-optimistic, and suggest that standard algorithm-choice scenarios will rarely lead to the best algorithm for individual instances in the space of interest. We do this by examining algorithm 'footprints', indicating how performance generalises in instance space. We find much evidence that typical ways of choosing the 'best' algorithm, via tests over a distribution of instances, are seriously flawed. Also, understanding how footprints in instance spaces vary between algorithms and across instance space dimensions, may lead to a future platform for wiser algorithm-choice decisions.

[1]  David Maxwell Chickering,et al.  A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report) , 2001, Electron. Notes Discret. Math..

[2]  Mauro Birattari,et al.  Tuning Metaheuristics - A Machine Learning Perspective , 2009, Studies in Computational Intelligence.

[3]  Henry A. Kautz,et al.  Auto-Walksat: A Self-Tuning Implementation of Walksat , 2001, Electron. Notes Discret. Math..

[4]  Ross J. W. James,et al.  A Knowledge Discovery Approach to Understanding Relationships between Scheduling Problem Structure and Heuristic Performance , 2009, LION.

[5]  Eugene S. Edgington,et al.  Randomization Tests , 2011, International Encyclopedia of Statistical Science.

[6]  Jonathan Gratch,et al.  Adaptive Problem-solving for Large-scale Scheduling Problems: A Case Study , 1996, J. Artif. Intell. Res..

[7]  Thomas Stützle,et al.  Automatic Algorithm Configuration Based on Local Search , 2007, AAAI.

[8]  David S. Johnson,et al.  A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[9]  Michael H. Goldwasser,et al.  Data Structures, Near Neighbor Searches, and Methodology: Fifth and Sixth DIMACS Implementation Challenges, Proceedings of a DIMACS Workshop, USA, 1999 , 2002, Data Structures, Near Neighbor Searches, and Methodology.

[10]  Thomas Bartz-Beielstein,et al.  Sequential parameter optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[11]  A. E. Eiben,et al.  A method for parameter calibration and relevance estimation in evolutionary algorithms , 2006, GECCO '06.

[12]  Steven Minton,et al.  Automatically configuring constraint satisfaction programs: A case study , 1996, Constraints.

[13]  Kevin Leyton-Brown,et al.  SATzilla: Portfolio-based Algorithm Selection for SAT , 2008, J. Artif. Intell. Res..

[14]  Thomas Stützle,et al.  Stochastic Local Search: Foundations & Applications , 2004 .

[15]  F. Hutter,et al.  ParamILS: an automatic algorithm configuration framework , 2009 .

[16]  J. Christopher Beck,et al.  APPLYING MACHINE LEARNING TO LOW‐KNOWLEDGE CONTROL OF OPTIMIZATION ALGORITHMS , 2005, Comput. Intell..

[17]  A. E. Eiben,et al.  Efficient relevance estimation and value calibration of evolutionary algorithm parameters , 2007, 2007 IEEE Congress on Evolutionary Computation.

[18]  Kevin Leyton-Brown,et al.  Performance Prediction and Automated Tuning of Randomized and Parametric Algorithms , 2006, CP.

[19]  Mike Preuss.  Adaptability of Algorithms for Real-Valued Optimization , 2009, EvoWorkshops.