To Measure or not to Measure? Adaptive Repetition Management in Parameter Tuning

Variance is an inherent part of manifold real-world empirical studies. It can appear due to non-deterministic behavior, measurement or systematic errors, information bias, etc. The former two causes are especially important for parameter tuning, as they can lead to an inaccurate search space estimation and a subsequent decrease in quality of the best found solution. Under parameter tuning we understand an optimization problem of determining the best configuration setting, given a system under investigation and a set of its configurations. Repetitive configuration evaluation is a standard approach to tackle the problem of variance. However, a high number of repetitions can overwhelmingly bloat an effort spent on the search, motivating the development of repetition management strategies. Nevertheless, numerous parameter tuning approaches are either neglecting this problem or suggesting repetition management strategies that are based on a predefined number of repetitions, which can vary for different problem instances. In this paper, we present a repetition management strategy that is inspired by the notion of acceptable measurement error to adaptively determine the number of repetitions for each configuration. Moreover, we investigate the influence of the knowledge on already performed measurements as the additional feedback for the presented repetition management strategy. We evaluate the effectiveness of our approaches with a set of energy optimization problems. The presented strategies prove to be more scalable and robust to changes, while the utilization of experiment-related knowledge can even further increase the quality of the found solution.

[1]  Panos Vassiliadis,et al.  A Survey of Extract-Transform-Load Technology , 2009, Int. J. Data Warehous. Min..

[2]  Derek Bingham,et al.  Handbook of Design and Analysis of Experiments , 2015 .

[3]  Peter Rossmanith,et al.  Simulated Annealing , 2008, Taschenbuch der Algorithmen.

[4]  Manuel López-Ibáñez,et al.  Ant colony optimization , 2010, GECCO '10.

[5]  Taimoor Akhtar,et al.  Efficient Multi-Objective Optimization through Population-based Parallel Surrogate Search , 2019, ArXiv.

[6]  Joshua D. Knowles,et al.  ParEGO: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems , 2006, IEEE Transactions on Evolutionary Computation.

[7]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[8]  Aaron Klein,et al.  BOHB: Robust and Efficient Hyperparameter Optimization at Scale , 2018, ICML.

[9]  Lars Schmidt-Thieme,et al.  Hyperparameter Optimization Machines , 2016, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[10]  Ke Tang,et al.  Evolutionary optimization with hierarchical surrogates , 2019, Swarm Evol. Comput..

[11]  M Dorigo,et al.  Ant colonies for the travelling salesman problem. , 1997, Bio Systems.

[12]  Thomas Bäck,et al.  Evolutionary algorithms in theory and practice - evolution strategies, evolutionary programming, genetic algorithms , 1996 .

[13]  Jean-Yves Potvin,et al.  Genetic Algorithms for the Traveling Salesman Problem , 2005 .

[14]  Sebastian Götz,et al.  BRISE: Energy-Efficient Benchmark Reduction , 2018, 2018 IEEE/ACM 6th International Workshop on Green And Sustainable Software (GREENS).

[15]  Thomas Bartz-Beielstein,et al.  Sequential parameter optimization , 2005, 2005 IEEE Congress on Evolutionary Computation.

[16]  Student,et al.  THE PROBABLE ERROR OF A MEAN , 1908 .

[17]  Uwe Aßmann,et al.  Parameter Tuning for Self-optimizing Software at Scale , 2019, ArXiv.

[18]  A. E. Eiben,et al.  Comparing parameter tuning methods for evolutionary algorithms , 2009, 2009 IEEE Congress on Evolutionary Computation.

[19]  Xin Liu,et al.  ACTGAN: Automatic Configuration Tuning for Software Systems with Generative Adversarial Networks , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[20]  A. E. Eiben,et al.  Multi-Problem Parameter Tuning using BONESA , 2011 .

[21]  Aaron Klein,et al.  Hyperparameter Optimization , 2017, Encyclopedia of Machine Learning and Data Mining.

[22]  Gordon Fraser,et al.  On Parameter Tuning in Search Based Software Engineering , 2011, SSBSE.

[23]  William T. Shaw,et al.  Sampling Student’s T distribution – use of the inverse cumulative distribution function , 2006 .

[24]  D. Streiner.  Maintaining Standards: Differences between the Standard Deviation and Standard Error, and When to Use Each , 1996, Canadian journal of psychiatry. Revue canadienne de psychiatrie.

[25]  Tom M. Mitchell,et al.  Learning by experimentation: acquiring and refining problem-solving heuristics , 1993 .

[26]  Graham Kendall,et al.  Hyper-Heuristics: An Emerging Direction in Modern Search Technology , 2003, Handbook of Metaheuristics.

[27]  A. E. Eiben,et al.  An MOEA-based Method to Tune EA Parameters on Multiple Objective Functions , 2010, IJCCI.

[28]  Handing Wang,et al.  Data-Driven Surrogate-Assisted Multiobjective Evolutionary Optimization of a Trauma System , 2016, IEEE Transactions on Evolutionary Computation.

[29]  M. Hartmann,et al.  A comparison of the performances of procedures for selecting the normal population having the largest mean when the populations have a common unknown variance , 1990 .

[30]  Palden Lama,et al.  AROMA: automated resource allocation and configuration of mapreduce environment in the cloud , 2012, ICAC '12.

[31]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[32]  Felipe Campelo,et al.  Tuning metaheuristics by sequential optimization of regression models , 2018, Appl. Soft Comput..

[33]  D. Sculley,et al.  Google Vizier: A Service for Black-Box Optimization , 2017, KDD.

[34]  Oliver Stegle,et al.  Gaussian Process Robust Regression for Noisy Heart Rate Data , 2008, IEEE Transactions on Biomedical Engineering.