ACTGAN: Automatic Configuration Tuning for Software Systems with Generative Adversarial Networks

Complex software systems often provide a large number of parameters so that users can configure them for their specific application scenarios. However, configuration tuning requires a deep understanding of the software system, far beyond the abilities of typical system users. To address this issue, many existing approaches focus on exploring and learning good performance estimation models. The accuracy of such models often suffers when the number of available samples is small, a thorny challenge under a given tuning-time constraint. By contrast, we hypothesize that good configurations often share certain hidden structures. Therefore, instead of trying to improve the performance estimation of a given configuration, we focus on capturing the hidden structures of good configurations and utilizing such learned structure to generate potentially better configurations. We propose ACTGAN to achieve this goal. We have implemented and evaluated ACTGAN using 17 workloads with eight different software systems. Experimental results show that ACTGAN outperforms default configurations by 76.22% on average, and six state-of-the-art configuration tuning algorithms by 6.58%-64.56%. Furthermore, the ACTGAN-generated configurations are often better than those used in training and show certain features consisting with domain knowledge, both of which supports our hypothesis.

[1]  Kenneth Steiglitz,et al.  Combinatorial Optimization: Algorithms and Complexity , 1981 .

[2]  Tao Ye,et al.  A recursive random search algorithm for large-scale network parameter configuration , 2003, SIGMETRICS '03.

[3]  Christian Blum,et al.  Metaheuristics in combinatorial optimization: Overview and conceptual comparison , 2003, CSUR.

[4]  Paola Inverardi,et al.  Model-based performance prediction in software development: a survey , 2004, IEEE Transactions on Software Engineering.

[5]  Bowei Xi,et al.  A smart hill-climbing algorithm for application server configuration , 2004, WWW '04.

[6]  Dorina C. Petriu,et al.  The Future of Software Performance Engineering , 2007, Future of Software Engineering (FOSE '07).

[7]  Heiko Koziolek,et al.  Performance evaluation of component-based software systems: A survey , 2010, Perform. Evaluation.

[8]  Jie Huang,et al.  The HiBench benchmark suite: Characterization of the MapReduce-based data analysis , 2010, 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010).

[9]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[10]  Wan Mohd. Nasir Wan Kadir,et al.  Comparative Analysis of Software Performance Prediction Approaches in Context of Component-based System , 2011 .

[11]  Liang Dong,et al.  Starfish: A Self-tuning System for Big Data Analytics , 2011, CIDR.

[12]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[13]  Marek Hatala,et al.  Automated planning for feature model configuration based on stakeholders' business concerns , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[14]  Haifeng Chen,et al.  Experience Transfer for the Configuration Tuning in Large-Scale Computing Systems , 2009, IEEE Transactions on Knowledge and Data Engineering.

[15]  Albert Y. Zomaya,et al.  On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments , 2012, ICA3PP.

[16]  Palden Lama,et al.  AROMA: automated resource allocation and configuration of mapreduce environment in the cloud , 2012, ICAC '12.

[17]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[18]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[19]  Xiaodong Li,et al.  Cooperatively Coevolving Particle Swarms for Large Scale Optimization , 2012, IEEE Transactions on Evolutionary Computation.

[20]  David D. Cox,et al.  Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms , 2013, SciPy.

[21]  Mark Harman,et al.  Searching for better configurations: a rigorous approach to clone evaluation , 2013, ESEC/FSE 2013.

[22]  Derek Rayside,et al.  Comparison of exact and approximate multi-objective optimization for software product lines , 2014, SPLC.

[23]  Yi Zhang,et al.  Performance Prediction of Configurable Software Systems by Fourier Learning (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[24]  Long Jin,et al.  Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software , 2015, ESEC/SIGSOFT FSE.

[25]  Fan Wu,et al.  Deep Parameter Optimisation , 2015, GECCO.

[26]  Deze Zeng,et al.  MR-COF: A Genetic MapReduce Configuration Optimization Framework , 2015, ICA3PP.

[27]  Kewen Wang,et al.  Performance Prediction for Apache Spark Platform , 2015, 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems.

[28]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[29]  Shane McIntosh,et al.  Automated Parameter Optimization of Classification Techniques for Defect Prediction Models , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[30]  Chengzhong Xu,et al.  Performance Modeling for Spark Using SVM , 2016, 2016 7th International Conference on Cloud Computing and Big Data (CCBD).

[31]  Fast Inference in Nonlinear Dynamical Systems using Gradient Matching , 2016 .

[32]  Yang Liu,et al.  Generating Performance Distributions via Probabilistic Symbolic Execution , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[33]  Lieven Eeckhout,et al.  RFHOC: A Random-Forest Approach to Auto-Tuning Hadoop's Configuration , 2016, IEEE Transactions on Parallel and Distributed Systems.

[34]  Norbert Siegmund,et al.  Transfer learning for performance modeling of configurable systems: An exploratory analysis , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[35]  Yuqing Zhu,et al.  BestConfig: tapping the performance potential of systems via automatic configuration tuning , 2017, SoCC.

[36]  Chong Tang System performance optimization via design and configuration space exploration , 2017, ESEC/SIGSOFT FSE.

[37]  Cheng Peng,et al.  A reinforcement learning approach to map reduce auto-configuration under networked environment , 2017, Int. J. Secur. Networks.

[38]  Sven Apel,et al.  Faster discovery of faster system configurations with spectral learning , 2017, Automated Software Engineering.

[39]  Don S. Batory,et al.  Finding near-optimal configurations in product lines by random sampling , 2017, ESEC/SIGSOFT FSE.

[40]  Sven Apel,et al.  Using bad learners to find good configurations , 2017, ESEC/SIGSOFT FSE.

[41]  Sven Apel,et al.  Data-efficient performance learning for configurable systems , 2018, Empirical Software Engineering.

[42]  Xin Liu,et al.  AutoConfig: Automatic Configuration Tuning for Distributed Message Systems , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[43]  Aaron Klein,et al.  BOHB: Robust and Efficient Hyperparameter Optimization at Scale , 2018, ICML.

[44]  Helmut Krcmar,et al.  A Workload-Dependent Performance Analysis of an In-Memory Database in a Multi-Tenant Configuration , 2018, ICPE Companion.

[45]  Shu Wang,et al.  Understanding and Auto-Adjusting Performance-Sensitive Configurations , 2018, ASPLOS.

[46]  Peter I. Frazier,et al.  A Tutorial on Bayesian Optimization , 2018, ArXiv.