Transferring Performance Prediction Models Across Different Hardware Platforms

Many software systems provide configuration options relevant to users, which are often called features. Features influence functional properties of software systems as well as non-functional ones, such as performance and memory consumption. Researchers have successfully demonstrated the correlation between feature selection and performance. However, the generality of these performance models across different hardware platforms has not yet been evaluated. We propose a technique for enhancing generality of performance models across different hardware environments using linear transformation. Empirical studies on three real-world software systems show that our approach is computationally efficient and can achieve high accuracy (less than 10% mean relative error) when predicting system performance across 23 different hardware platforms. Moreover, we investigate why the approach works by comparing performance distributions of systems and structure of performance models across different platforms.

[1]  Kevin Leyton-Brown,et al.  Algorithm runtime prediction: Methods & evaluation , 2012, Artif. Intell..

[2]  Brian D. Ripley,et al.  Stochastic Simulation , 2005 .

[3]  Sven Apel,et al.  Variability-aware performance prediction: A statistical learning approach , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[4]  Krzysztof Czarnecki,et al.  Empirical comparison of regression methods for variability-aware performance prediction , 2015, SPLC.

[5]  James J. Filliben,et al.  NIST/SEMATECH e-Handbook of Statistical Methods; Chapter 1: Exploratory Data Analysis , 2003 .

[6]  Mariano Ruiz Espejo,et al.  Design of Experiments for Engineers and Scientists , 2006, Technometrics.

[7]  Simonetta Balsamo,et al.  Performance evaluation of UML software architectures with multiclass Queueing Network models , 2005, WOSP '05.

[8]  Vittorio Cortellessa,et al.  PRIMA-UML: a performance validation incremental methodology on early UML diagrams , 2002, Sci. Comput. Program..

[9]  Peter Nobel,et al.  Practical performance models for complex, popular applications , 2010, SIGMETRICS '10.

[10]  Ina Schaefer,et al.  Scaling Size and Parameter Spaces in Variability-Aware Software Performance Models (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[11]  Lieven Eeckhout,et al.  Performance prediction based on inherent program similarity , 2006, 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[12]  Yi Zhang,et al.  Performance Prediction of Configurable Software Systems by Fourier Learning (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[13]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[14]  William H. Press,et al.  Numerical recipes in C (2nd ed.): the art of scientific computing , 1992 .

[15]  I. Sobol,et al.  A pseudo-random number generator for personal computers , 1999 .

[16]  Y. Zhang,et al.  DataMill: a distributed heterogeneous infrastructure forrobust experimentation , 2016, Softw. Pract. Exp..

[17]  Rouven Krebs,et al.  Automated inference of goal-oriented performance prediction functions , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[18]  Kevin Leyton-Brown,et al.  Algorithm Runtime Prediction: Methods and Evaluation (Extended Abstract) , 2015, IJCAI.