Towards objective measures of algorithm performance across instance space

This paper tackles the difficult but important task of objective algorithm performance assessment for optimization. Rather than reporting average performance of algorithms across a set of chosen instances, which may bias conclusions, we propose a methodology to enable the strengths and weaknesses of different optimization algorithms to be compared across a broader instance space. The results reported in a recent Computers and Operations Research paper comparing the performance of graph coloring heuristics are revisited with this new methodology to demonstrate (i) how pockets of the instance space can be found where algorithm performance varies significantly from the average performance of an algorithm; (ii) how the properties of the instances can be used to predict algorithm performance on previously unseen instances with high accuracy; and (iii) how the relative strengths and weaknesses of each algorithm can be visualized and measured objectively.

[1]  Eoin O'Mahony,et al.  Using Case-based Reasoning in an Algorithm Portfolio for Constraint Solving ? , 2008 .

[2]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[3]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[4]  A. Vázquez,et al.  Network clustering coefficient without degree-correlation biases. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[5]  Joseph C. Culberson,et al.  On the Futility of Blind Search: An Algorithmic View of No Free Lunch , 1998, Evolutionary Computation.

[6]  Joseph Culberson.  On the Futility of Blind Search , 1996 .

[7]  Jano I. van Hemert,et al.  Discovering the suitability of optimisation algorithms by learning from evolved instances , 2011, Annals of Mathematics and Artificial Intelligence.

[8]  D. de Werra,et al.  An introduction to timetabling , 1985 .

[9]  David S. Johnson,et al.  Cliques, Coloring, and Satisfiability , 1996 .

[10]  Charles H. Reilly,et al.  The Effects of Coefficient Correlation Structure in Two-Dimensional Knapsack Problems on Solution Procedure Performance , 2000 .

[11]  Cecilia R. Aragon,et al.  Optimization by Simulated Annealing: An Experimental Evaluation; Part II, Graph Coloring and Number Partitioning , 1991, Oper. Res..

[12]  Rhyd Lewis,et al.  A survey of metaheuristic-based techniques for University Timetabling problems , 2007, OR Spectr..

[13]  Daniel Brélaz,et al.  New methods to color the vertices of a graph , 1979, CACM.

[14]  Hadrien Mélot,et al.  House of Graphs: A database of interesting graphs , 2012, Discret. Appl. Math..

[15]  R. Balakrishnan.  The energy of a graph , 2004 .

[16]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[17]  Jano I. van Hemert,et al.  Understanding TSP Difficulty by Learning from Evolved Instances , 2010, LION.

[18]  Kate Smith-Miles,et al.  Towards insightful algorithm selection for optimisation using meta-learning concepts , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[19]  B. Mohar.  THE LAPLACIAN SPECTRUM OF GRAPHS y , 1991 .

[20]  Nicolas Chapados,et al.  Extensions to Metric-Based Model Selection , 2003, J. Mach. Learn. Res..

[21]  Jürgen Schmidhuber,et al.  Learning dynamic algorithm portfolios , 2006, Annals of Mathematics and Artificial Intelligence.

[22]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[23]  Hadrien Mélot,et al.  Facet defining inequalities among graph invariants: The system GraPHedron , 2008, Discret. Appl. Math..

[24]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[25]  T. Kohonen.  Self-organized formation of topographically correct feature maps , 1982 .

[26]  Dominique de Werra,et al.  Some models of graphs for scheduling sports competitions , 1988, Discret. Appl. Math..

[27]  Jing Ma,et al.  The Cunningham-Geelen Method in Practice: Branch-Decompositions and Integer Programming , 2013, INFORMS J. Comput..

[28]  D. R. White,et al.  Structural cohesion and embeddedness: A hierarchical concept of social groups , 2003 .

[29]  Teuvo Kohonen,et al.  Self-organized formation of topologically correct feature maps , 2004, Biological Cybernetics.

[30]  M. Trick,et al.  Cliques, Coloring, and Satisfiability: Second DIMACS Implementation Challenge, Workshop, October 11-13, 1993 , 1996 .

[31]  I. Jolliffe.  Principal Component Analysis , 2002 .

[32]  Kate Smith-Miles,et al.  Generating Applicable Synthetic Instances for Branch Problems , 2013, Oper. Res..

[33]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[34]  Sanja Petrovic,et al.  A graph-based hyper-heuristic for educational timetabling problems , 2007, Eur. J. Oper. Res..

[35]  Norman Biggs.  Algebraic Graph Theory: Index , 1974 .

[36]  Kate Smith-Miles,et al.  Cross-disciplinary perspectives on meta-learning for algorithm selection , 2009, CSUR.

[37]  Rhyd Lewis,et al.  A general-purpose hill-climbing method for order independent minimum grouping problems: A case study in graph colouring and bin packing , 2009, Comput. Oper. Res..

[38]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[39]  John R. Rice,et al.  The Algorithm Selection Problem , 1976, Adv. Comput..

[40]  Robin J. Wilson,et al.  An Atlas of Graphs , 1999 .

[41]  Kate Smith-Miles,et al.  Exploring the role of graph spectra in graph coloring algorithm performance , 2014, Discret. Appl. Math..

[42]  Tomaz Pisanski,et al.  Use of the Szeged index and the revised Szeged index for measuring network bipartivity , 2010, Discret. Appl. Math..

[43]  John E. Beasley,et al.  OR-Library: Distributing Test Problems by Electronic Mail , 1990 .

[44]  Jin-Kao Hao,et al.  Hybrid Evolutionary Algorithms for Graph Coloring , 1999, J. Comb. Optim..

[45]  LewisRhyd.  A general-purpose hill-climbing method for order independent minimum grouping problems , 2009 .

[46]  John R. Rice,et al.  PYTHIA: a knowledge-based system to select scientific algorithms , 1996, TOMS.

[47]  Jonathan M. Thompson,et al.  A wide-ranging computational comparison of high-performance graph colouring algorithms , 2012, Comput. Oper. Res..

[48]  David R. Wood,et al.  An algorithm for finding a maximum clique in a graph , 1997, Oper. Res. Lett..

[49]  Kathryn A. Dowsland,et al.  An improved ant colony optimisation heuristic for graph colouring , 2008, Discret. Appl. Math..

[50]  Gilbert Laporte,et al.  Examination Timetabling: Algorithmic Strategies and Applications , 1994 .

[51]  Kate Smith-Miles,et al.  Predicting Metaheuristic Performance on Graph Coloring Problems Using Data Mining , 2013, Hybrid Metaheuristics.

[52]  Kate Smith-Miles,et al.  Measuring algorithm footprints in instance space , 2012, 2012 IEEE Congress on Evolutionary Computation.

[53]  Kate Smith-Miles,et al.  Measuring instance difficulty for combinatorial optimization problems , 2012, Comput. Oper. Res..

[54]  Kevin Leyton-Brown,et al.  : The Design and Analysis of an Algorithm Portfolio for SAT , 2007, CP.

[55]  Marc Toussaint,et al.  A No-Free-Lunch Theorem for Non-Uniform Distributions of Target Functions , 2004 .

[56]  Kate Smith-Miles,et al.  Generalising Algorithm Performance in Instance Space: A Timetabling Case Study , 2011, LION.

[57]  Ross J. W. James,et al.  A Knowledge Discovery Approach to Understanding Relationships between Scheduling Problem Structure and Heuristic Performance , 2009, LION.

[58]  J. A. Rodríguez-Velázquez,et al.  Spectral measures of bipartivity in complex networks. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[59]  Naren Ramakrishnan,et al.  GAUSS: an online algorithm selection system for numerical quadrature , 2002 .

[60]  Nicolas Zufferey,et al.  A graph coloring heuristic using partial solutions and a reactive tabu scheme , 2008, Comput. Oper. Res..

[61]  Padraig Cunningham,et al.  Diversity in Ensemble Feature Selection , 2003 .

[62]  Cecilia R. Aragon,et al.  Optimization by Simulated Annealing: An Experimental Evaluation; Part I, Graph Partitioning , 1989, Oper. Res..