An Empirical Study on Performance Bugs for Highly Configurable Software Systems

Modern computer systems are highly-configurable, complicating the testing and debugging process. The sheer size of the configuration space makes the quality of software even harder to achieve. Performance is one of the key aspects of non-functional qualities, where performance bugs can cause significant performance degradation and lead to poor user experience. However, performance bugs are difficult to expose, primarily because detecting them requires specific inputs, as well as a specific execution environment (e.g., configurations). While researchers have developed techniques to analyze, quantify, detect, and fix performance bugs, we conjecture that many of these techniques may not be effective in highly-configurable systems. In this paper, we study the challenges that configurability creates for handling performance bugs. We study 113 real-world performance bugs, randomly sampled from three highly-configurable open-source projects: Apache, MySQL and Firefox. The findings of this study provide a set of lessons learned and guidance to aid practitioners and researchers to better handle performance bugs in highly-configurable software systems.

[1]  Mona Attariyan,et al.  X-ray: Automating Root-Cause Diagnosis of Performance Anomalies in Production Software , 2012, OSDI.

[2]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[3]  Isil Dillig,et al.  Static detection of asymptotic performance bugs in collection traversals , 2015, PLDI.

[4]  Shan Lu,et al.  Toddler: Detecting performance problems via similar memory-access patterns , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[5]  Matthias Hauswirth,et al.  Catch me if you can: performance bug detection in the wild , 2011, OOPSLA '11.

[6]  Koushik Sen,et al.  WISE: Automated test generation for worst-case complexity , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[7]  Gina Venolia,et al.  The secret life of bugs: Going past the errors and omissions in software repositories , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[8]  Sven Apel,et al.  Variability-aware performance prediction: A statistical learning approach , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[9]  Shan Lu,et al.  Statistical debugging for real-world performance problems , 2014, OOPSLA.

[10]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[11]  Mona Attariyan,et al.  Automating Configuration Troubleshooting with Dynamic Information Flow Analysis , 2010, OSDI.

[12]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[13]  Dongmei Zhang,et al.  Performance debugging in the large via mining millions of stack traces , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[14]  Myra B. Cohen,et al.  Beyond the rainbow: self-adaptive failure avoidance in configurable systems , 2014, SIGSOFT FSE.

[15]  Chen Fu,et al.  Automatically finding performance problems with feedback-directed learning software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[16]  Nelly Condori-Fernandez,et al.  Evaluating applicability of combinatorial testing in an industrial environment: a case study , 2013, JAMAICA 2013.

[17]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[18]  Michael D. Ernst,et al.  Automated diagnosis of software configuration errors , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[19]  Tian Jiang,et al.  Discovering, reporting, and fixing performance bugs , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[20]  Premkumar T. Devanbu,et al.  Fair and balanced?: bias in bug-fix datasets , 2009, ESEC/FSE '09.

[21]  Ying Zou,et al.  An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[22]  Eric Bodden,et al.  Tracking Load-Time Configuration Options , 2014, IEEE Transactions on Software Engineering.

[23]  Yunheung Paek,et al.  Mantis: Automatic Performance Prediction for Smartphone Applications , 2013, USENIX Annual Technical Conference.

[24]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[25]  Steven P. Reiss,et al.  Automated analysis of multithreaded programs for performance modeling , 2014, SIGMETRICS '14.

[26]  Randy H. Katz,et al.  Static extraction of program configuration options , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[27]  Matthew Arnold,et al.  Software bloat analysis: finding, removing, and preventing performance problems in modern large-scale object-oriented applications , 2010, FoSER '10.

[28]  Jens Happe,et al.  Supporting swift reaction: Automatically uncovering performance problems by systematic experiments , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[29]  Avelino Francisco Zorzo,et al.  Evaluating capture and replay and model-based performance testing tools: an empirical comparison , 2014, ESEM '14.

[30]  Sven Apel,et al.  Performance-influence models for highly configurable systems , 2015, ESEC/SIGSOFT FSE.

[31]  Adam A. Porter,et al.  Using symbolic evaluation to understand behavior in configurable software systems , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[32]  Myra B. Cohen,et al.  Configurations everywhere: implications for testing and debugging in practice , 2014, ICSE Companion.

[33]  Rainer Koschke,et al.  Locating Features in Source Code , 2003, IEEE Trans. Software Eng..

[34]  Ahmed E. Hassan,et al.  A qualitative study on performance bugs , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[35]  Thomas R. Gross,et al.  Performance regression testing of concurrent classes , 2014, ISSTA 2014.

[36]  Xiao Ma,et al.  An empirical study on configuration errors in commercial and open source systems , 2011, SOSP.