LONViZ: Unboxing the black-box of Configurable Software Systems from a Complex Networks Perspective

Abstract: Most, if not all, modern software systems are highly configurable to tailor both their functional and non-functional properties to a variety of stakeholders. Due to the black-box nature, it is difficult, if not impossible, to analyze and understand its behavior, such as the interaction between combinations of configuration options with regard to the performance, in particular, which is of great importance to advance the controllability of the underlying software system. This paper proposes a tool, dubbed LONViZ, which is the first of its kind, to facilitate the exploratory analysis of black-box configurable software systems. It starts from a systematic sampling over the configuration space of the underlying system. Then LONViZ seeks to construct a structurally stable LON by synthesizing multiple repeats of sampling results. Finally, exploratory analysis can be conducted on the stable LON from both qualitative and quantitative perspectives. In experiments, we choose four widely used real-world configurable software systems to develop benchmark platforms under 42 different running environments. From our empirical study, we find that LONViZ enables both qualitative and quantitative analysis and disclose various interesting hidden patterns and properties of different software systems.

[1]  Gabriela Ochoa,et al.  Recent advances in fitness landscape analysis , 2019, GECCO.

[2]  L. Darrell Whitley,et al.  Tunnelling Crossover Networks , 2015, GECCO.

[3]  Christian Kästner,et al.  Transfer Learning for Improving Model Predictions in Highly Configurable Software , 2017, 2017 IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[4]  Sven Apel,et al.  Performance-influence models for highly configurable systems , 2015, ESEC/SIGSOFT FSE.

[5]  Qiuzhen Lin,et al.  Vertical distance-based clonal selection mechanism for the multiobjective immune algorithm , 2021, Swarm Evol. Comput..

[6]  Sam Kwong,et al.  Class-specific soft voting based multiple extreme learning machines ensemble , 2015, Neurocomputing.

[7]  Shengxiang Yang,et al.  A knee-point-based evolutionary algorithm using weighted subpopulation for many-objective optimization , 2019, Swarm Evol. Comput..

[8]  Qingfu Zhang,et al.  Evolutionary Many-Objective Optimization Based on Adversarial Decomposition , 2017, IEEE Transactions on Cybernetics.

[9]  Sam Kwong,et al.  EVOLVING EXTREME LEARNING MACHINE PARADIGM WITH ADAPTIVE OPERATOR SELECTION AND PARAMETER CONTROL , 2013 .

[10]  Qingfu Zhang,et al.  Decomposition multi-objective optimisation: current developments and future opportunities , 2018, GECCO.

[11]  Michael Affenzeller,et al.  A Comprehensive Survey on Fitness Landscape Analysis , 2012, Recent Advances in Intelligent Engineering Systems.

[12]  Marco Tomassini,et al.  Understanding Phase Transitions with Local Optima Networks: Number Partitioning as a Case Study , 2017, EvoCOP.

[13]  Geyong Min,et al.  A Formal Model for Multi-objective Optimisation of Network Function Virtualisation Placement , 2019, EMO.

[14]  Kay Chen Tan,et al.  Which Surrogate Works for Empirical Performance Modelling? A Case Study with Differential Evolution , 2019, 2019 IEEE Congress on Evolutionary Computation (CEC).

[15]  Don S. Batory,et al.  Finding near-optimal configurations in product lines by random sampling , 2017, ESEC/SIGSOFT FSE.

[16]  Cong Zhou,et al.  A novel algorithm for non-dominated hypervolume-based multiobjective optimization , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[17]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[18]  M E J Newman.  Assortative mixing in networks. , 2002, Physical review letters.

[19]  Gerhard Nahler,et al.  Pearson Correlation Coefficient , 2020, Definitions.

[20]  Sam Kwong,et al.  Bayesian Network Based Label Correlation Analysis For Multi-label Classifier Chain , 2019, Inf. Sci..

[21]  J. Doye,et al.  THE DOUBLE-FUNNEL ENERGY LANDSCAPE OF THE 38-ATOM LENNARD-JONES CLUSTER , 1998, cond-mat/9808265.

[22]  Tao Chen,et al.  DeepSQLi: deep semantic learning for testing SQL injection , 2020, ISSTA.

[23]  Kay Chen Tan,et al.  BiLO-CPDP: Bi-Level Programming for Automated Model Discovery in Cross-Project Defect Prediction , 2020, 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[24]  Gabriela Ochoa,et al.  Local optima networks for continuous fitness landscapes , 2019, GECCO.

[25]  Álvaro Fialho,et al.  Multi-Objective Differential Evolution with Adaptive Control of Parameters and Operators , 2011, LION.

[26]  Sam Kwong,et al.  A general framework for evolutionary multiobjective optimization via manifold learning , 2014, Neurocomputing.

[27]  Xin Yao,et al.  Does Preference Always Help? A Holistic Study on Preference-Based Evolutionary Multiobjective Optimization Using Reference Points , 2019, IEEE Transactions on Evolutionary Computation.

[28]  Christian Kästner,et al.  Learning to sample: exploiting similarities across environments to learn performance models for configurable systems , 2018, ESEC/SIGSOFT FSE.

[29]  Derek Rayside,et al.  Comparison of exact and approximate multi-objective optimization for software product lines , 2014, SPLC.

[30]  Xin Yao,et al.  Dynamic Multiobjectives Optimization With a Changing Number of Objectives , 2016, IEEE Transactions on Evolutionary Computation.

[31]  Ke Li,et al.  Visualisation of Pareto Front Approximation: A Short Survey and Empirical Comparisons , 2019, 2019 IEEE Congress on Evolutionary Computation (CEC).

[32]  Sven Apel,et al.  White-Box Analysis over Machine Learning: Modeling Performance of Configurable Systems , 2021, 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE).

[33]  Sven Apel,et al.  ConfigCrusher: towards white-box performance analysis for configurable systems , 2020, Automated Software Engineering.

[34]  Sam Kwong,et al.  A weighted voting method using minimum square error based on Extreme Learning Machine , 2012, 2012 International Conference on Machine Learning and Cybernetics.

[35]  Satish Kumar,et al.  Multi-Tenant Cloud Service Composition Using Evolutionary Optimization , 2018, 2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS).

[36]  Qingfu Zhang,et al.  Evolutionary multiobjective optimization with hybrid selection principles , 2015, 2015 IEEE Congress on Evolutionary Computation (CEC).

[37]  Kim-Fung Man,et al.  Learning paradigm based on jumping genes: A general framework for enhancing exploration in evolutionary multiobjective optimization , 2013, Inf. Sci..

[38]  Kalyanmoy Deb,et al.  A dual-population paradigm for evolutionary multiobjective optimization , 2015, Inf. Sci..

[39]  Sam Kwong,et al.  AN indicator-based selection multi-objective evolutionary algorithm with preference for multi-class ensemble , 2014, 2014 International Conference on Machine Learning and Cybernetics.

[40]  Yuanyuan Zhou,et al.  Do not blame users for misconfigurations , 2013, SOSP.

[41]  Sam Kwong,et al.  Multi-objective differential evolution with self-navigation , 2012, 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[42]  Xin Yao,et al.  Empirical Investigations of Reference Point Based Methods When Facing a Massively Large Number of Objectives: First Results , 2017, EMO.

[43]  Yang Liu,et al.  graph2vec: Learning Distributed Representations of Graphs , 2017, ArXiv.

[44]  Gabriela Ochoa,et al.  Mapping the global structure of TSP fitness landscapes , 2017, J. Heuristics.

[45]  Qingfu Zhang,et al.  Two-Level Stable Matching-Based Selection in MOEA/D , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[46]  Krzysztof Czarnecki,et al.  Transferring Performance Prediction Models Across Different Hardware Platforms , 2017, ICPE.

[47]  Sébastien Vérel,et al.  A study of NK landscapes' basins and local optima networks , 2008, GECCO '08.

[48]  Xin Yao,et al.  R-Metric: Evaluating the Performance of Preference-Based Evolutionary Multiobjective Optimization Using Reference Points , 2018, IEEE Transactions on Evolutionary Computation.

[49]  Yuqing Zhu,et al.  BestConfig: tapping the performance potential of systems via automatic configuration tuning , 2017, SoCC.

[50]  Krzysztof Czarnecki,et al.  Empirical comparison of regression methods for variability-aware performance prediction , 2015, SPLC.

[51]  Andries P. Engelbrecht,et al.  Recent Advances in the Theory and Application of Fitness Landscapes , 2013 .

[52]  Gabriela Ochoa,et al.  Modelling parameter configuration spaces with local optima networks , 2020, GECCO.

[53]  Thomas Stützle,et al.  Iterated local search for the quadratic assignment problem , 2006, Eur. J. Oper. Res..

[54]  Ke Li,et al.  Empirical Studies on the Role of the Decision Maker in Interactive Evolutionary Multi-Objective Optimization , 2021, 2021 IEEE Congress on Evolutionary Computation (CEC).

[55]  Kay Chen Tan,et al.  Understanding the Automated Parameter Optimization on Transfer Learning for Cross-Project Defect Prediction: An Empirical Study , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[56]  Gianluca Palermo,et al.  A Survey on Compiler Autotuning using Machine Learning , 2018, ACM Comput. Surv..

[57]  Sam Kwong,et al.  Combining interpretable fuzzy rule-based classifiers via multi-objective hierarchical evolutionary algorithm , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[58]  Alessandro Vespignani,et al.  Detecting rich-club ordering in complex networks , 2006, physics/0602134.

[59]  Qingfu Zhang,et al.  Adaptive weights generation for decomposition-based multi-objective optimization using Gaussian process regression , 2017, GECCO.

[60]  Tao Chen,et al.  Security testing of web applications: a search-based approach for detecting SQL injection vulnerabilities , 2019, GECCO.

[61]  Sven Apel,et al.  Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[62]  Sébastien Vérel,et al.  Sampling Local Optima Networks of Large Combinatorial Search Spaces: The QAP Case , 2018, PPSN.

[63]  Giuliano Casale,et al.  An Uncertainty-Aware Approach to Optimal Configuration of Stream Processing Systems , 2016, 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS).

[64]  Sven Apel,et al.  Finding Faster Configurations Using FLASH , 2018, IEEE Transactions on Software Engineering.

[65]  B. F. Rasmussen,et al.  A Topographic View of Supercooled Liquids and Glass Formation , 2005 .

[66]  Jinhua Zheng,et al.  Achieving balance between proximity and diversity in multi-objective evolutionary algorithm , 2012, Inf. Sci..

[67]  Qingfu Zhang,et al.  Efficient Nondomination Level Update Method for Steady-State Evolutionary Multiobjective Optimization , 2017, IEEE Transactions on Cybernetics.

[68]  Long Jin,et al.  Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software , 2015, ESEC/SIGSOFT FSE.

[69]  Gabriela Ochoa,et al.  Deconstructing the Big Valley Search Space Hypothesis , 2016, EvoCOP.

[70]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[71]  Tingting Yu,et al.  PerfLearner: Learning from Bug Reports to Understand and Generate Performance Test Frames , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[72]  Hendrik Richter.  Fitness Landscapes and Evolutionary Dynamics , 2012, NOSTRADAMUS.

[73]  Ke Li,et al.  An Improved Two-Archive Evolutionary Algorithm for Constrained Multi-Objective Optimization , 2021, EMO.

[74]  Ke Li,et al.  Progressive Preference Learning: Proof-of-Principle Results in MOEA/D , 2019, EMO.

[75]  Xin Yao,et al.  Interactive Decomposition Multiobjective Optimization Via Progressively Learned Value Functions , 2018, IEEE Transactions on Fuzzy Systems.

[76]  Cong Zhou,et al.  An Improved Differential Evolution for Multi-objective Optimization , 2009, 2009 WRI World Congress on Computer Science and Information Engineering.

[77]  Hongyu Zhang,et al.  DeepPerf: Performance Prediction for Configurable Software with Deep Sparse Neural Network , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).