Empirical scaling analyzer: An automated system for empirical analysis of performance scaling

The time complexity of algorithms, i.e., the scaling of the time required for solving a problem instance as a function of instance size, is of key interest in theoretical computer science and practical applications. In this work, we present a fully automated tool – Empirical Scaling Analyzer (ESA) – for performing sophisticated and detailed empirical scaling analyses. The methodological approach underlying ESA is based on a combination of automatic function fitting and bootstrap sampling; previous versions of the methodology have been used in prior work to characterize the empirical scaling behaviour of several prominent, high-performance SAT and TSP solvers. ESA is applicable to any algorithm or system, as long as running time data can be collected on sets of problem instances of various sizes. We present results from rigorous stress-testing to critically assess ESA on scenarios with challenging characteristics. We also give an overview of empirical scaling results obtained using ESA.

[1]  Thomas Stützle,et al.  On the empirical scaling of running time for finding optimal solutions to the TSP , 2018, J. Heuristics.

[2]  Thomas Stützle,et al.  The Impact of Automated Algorithm Configuration on the Scaling Behaviour of State-of-the-Art Inexact TSP Solvers , 2016, LION.

[3]  Aaron Klein,et al.  Efficient and Robust Automated Machine Learning , 2015, NIPS.

[4]  Holger H. Hoos,et al.  On the Empirical Time Complexity of Random 3-SAT at the Phase Transition , 2015, IJCAI.

[5]  Thomas Stützle,et al.  On the Empirical Scaling Behaviour of State-of-the-art Local Search Algorithms for the Euclidean TSP , 2015, GECCO.

[6]  Zongxu Mu.  Analysing the empirical time complexity of high-performance algorithms for SAT and TSP , 2015 .

[7]  Thomas Stützle,et al.  On the empirical scaling of run-time for finding optimal solutions to the travelling salesman problem , 2014, Eur. J. Oper. Res..

[8]  Camil Demetrescu,et al.  Input-Sensitive Profiling , 2012, IEEE Transactions on Software Engineering.

[9]  Holger H. Hoos,et al.  Ordered racing protocols for automatically configuring algorithms for scaling performance , 2013, GECCO '13.

[10]  Shigenobu Kobayashi,et al.  A Powerful Genetic Algorithm Using Edge Assembly Crossover for the Traveling Salesman Problem , 2013, INFORMS J. Comput..

[11]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[12]  Olaf Chitil,et al.  Practical typed lazy contracts , 2012, ICFP.

[13]  Matthias Hauswirth,et al.  Algorithmic profiling , 2012, PLDI.

[14]  Holger H. Hoos,et al.  Automatically Configuring Algorithms for Scaling Performance , 2012, LION.

[15]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[16]  Quan Sun,et al.  Sampling-based Prediction of Algorithm Runtime , 2009 .

[17]  William J. Cook,et al.  The Traveling Salesman Problem: A Computational Study , 2007 .

[18]  Anne Condon,et al.  Computational RNA secondary structure design: empirical complexity and improved methods , 2007, BMC Bioinformatics.

[19]  Gregory Gutin,et al.  The traveling salesman problem , 2006, Discret. Optim..

[20]  K. Subramani,et al.  On the Empirical Efficiency of the Vertex Contraction Algorithm for Detecting Negative Cost Cyles in Networks , 2005, International Conference on Computational Science.

[21]  Thomas Stützle,et al.  Stochastic Local Search: Foundations & Applications , 2004 .

[22]  Gilles Dequen,et al.  kcnfs: An Efficient Solver for Random k-SAT Formulae , 2003, SAT.

[23]  Daniel Kunkle.  Empirical Complexities of Longest Common Subsequence Algorithms , 2002 .

[24]  Keld Helsgaun,et al.  An effective implementation of the Lin-Kernighan traveling salesman heuristic , 2000, Eur. J. Oper. Res..

[25]  Paul R. Cohen,et al.  Using Finite Experiments to Study Asymptotic Performance , 2000, Experimental Algorithmics.

[26]  Thomas Stützle,et al.  Towards a Characterisation of the Behaviour of Stochastic Local Search Algorithms for SAT , 1999, Artif. Intell..

[27]  Shigenobu Kobayashi,et al.  Edge Assembly Crossover: A High-Power Genetic Algorithm for the Travelling Salesman Problem , 1997, ICGA.

[28]  Bart Selman,et al.  Noise Strategies for Improving Local Search , 1994, AAAI.