BRISE: Energy-Efficient Benchmark Reduction

A considerable portion of research activities in computer science heavily relies on the process of benchmarking, e.g., to evaluate a hypothesis in an empirical study. The goal is to reveal how a set of independent variables (factors) influences one or more dependent variables. With a vast number of factors or a high amount of factors' values (levels), this process becomes time- and energy-consuming. Current approaches to lower the benchmarking effort suffer from two deficiencies: (1) they focus on reducing the number of factors and, hence, are inapplicable to experiments with only two factors, but a vast number of levels and (2) being adopted from, e.g., combinatorial optimization they are designed for a different search space structure and, thus, can be very wasteful. This paper provides an approach for benchmark reduction, based on adaptive instance selection and multiple linear regression. We evaluate our approach using four empirical studies, which investigate the effect made by dynamic voltage and frequency scaling in combination with dynamic concurrency throttling on the energy consumption of a computing system (parallel compression, sorting, and encryption algorithms as well as database query processing). Our findings show the effectiveness of the approach. We can save 78% of benchmarking effort, while the result's quality decreases only by 3 pp, due to using only a near-optimal configuration.

[1]  Wolfgang Lehner,et al.  Energy-Efficient Databases Using Sweet Spot Frequencies , 2014, 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing.

[2]  F. Hutter,et al.  ParamILS: an automatic algorithm configuration framework , 2009 .

[3]  Kevin Leyton-Brown,et al.  Algorithm runtime prediction: Methods & evaluation , 2012, Artif. Intell..

[4]  Aaron Klein,et al.  Bayesian Optimization with Robust Bayesian Neural Networks , 2016, NIPS.

[5]  Yen-Sheng Chen,et al.  Development of a Compact LTE Dual-Band Antenna Using Fractional Factorial Design , 2015, IEEE Antennas and Wireless Propagation Letters.

[6]  Elaine B. Barker,et al.  SP 800-67 Rev. 1. Recommendation for the Triple Data Encryption Algorithm (TDEA) Block Cipher , 2004 .

[7]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[8]  Sven Apel,et al.  Performance-influence models for highly configurable systems , 2015, ESEC/SIGSOFT FSE.

[9]  Thomas Ilsche,et al.  An Energy Efficiency Feature Survey of the Intel Haswell Processor , 2015, 2015 IEEE International Parallel and Distributed Processing Symposium Workshop.

[10]  Vincent Rijmen,et al.  The WHIRLPOOL Hashing Function , 2003 .

[11]  Josef Spillner,et al.  Energy-Efficient Data Processing at Sweet Spot Frequencies , 2014, OTM Workshops.

[12]  Huan Liu,et al.  On Issues of Instance Selection , 2002, Data Mining and Knowledge Discovery.

[13]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[14]  Frederic P. Miller,et al.  Advanced Encryption Standard , 2009 .

[15]  Daniel G. Anderson,et al.  Optimization of Lipid Nanoparticle Formulations for mRNA Delivery in Vivo with Fractional Factorial and Definitive Screening Designs. , 2015, Nano letters.

[16]  S. Goldfeld,et al.  Maximization by Quadratic Hill-Climbing , 1966 .

[17]  I. Sobol.  On the distribution of points in a cube and the approximate evaluation of integrals , 1967 .

[18]  Derek Bingham,et al.  Handbook of Design and Analysis of Experiments , 2015 .

[19]  Weng Kee Wong,et al.  Application of fractional factorial designs to study drug combinations , 2013, Statistics in medicine.