CP-Detector: Using Configuration-related Performance Properties to Expose Performance Bugs

Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance (e.g., throughput, latency, execution time) expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects common expectation on performance changes. If the actual performance is contrary to the expectation, the related code snippet is likely to be problematic. In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as the test oracle in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-DETECTOR, for detecting real-world configuration-related performance bugs. CP-DETECTOR was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.

[1]  Camil Demetrescu,et al.  Input-Sensitive Profiling , 2012, IEEE Transactions on Software Engineering.

[2]  Matthias Hauswirth,et al.  Algorithmic profiling , 2012, PLDI.

[3]  Thomas R. Gross,et al.  Performance problems you can fix: a dynamic analysis of memoization opportunities , 2015, OOPSLA.

[4]  Don S. Batory,et al.  Finding near-optimal configurations in product lines by random sampling , 2017, ESEC/SIGSOFT FSE.

[5]  Andreas Zeller,et al.  Mining Version Histories to Guide Software Changes , 2004 .

[6]  Tingting Yu,et al.  PerfLearner: Learning from Bug Reports to Understand and Generate Performance Test Frames , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[7]  Amir Michail,et al.  Data mining library reuse patterns in user-selected applications , 1999, 14th IEEE International Conference on Automated Software Engineering.

[8]  Shriram K. Vasudevan,et al.  Dictionary of Computer Science , 2016 .

[9]  Shankar Pasupathy,et al.  PracExtractor: Extracting Configuration Good Practices from Manuals to Detect Server Misconfigurations , 2020, USENIX Annual Technical Conference.

[10]  Tingting Yu,et al.  An Empirical Study on Performance Bugs for Highly Configurable Software Systems , 2016, ESEM.

[11]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[12]  David S. Rosenblum,et al.  Mining performance specifications , 2016, SIGSOFT FSE.

[13]  Weiyi Shang,et al.  Towards the Use of the Readily Available Tests from the Release Pipeline as Performance Tests. Are We There Yet? , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[14]  Shan Lu,et al.  Toddler: Detecting performance problems via similar memory-access patterns , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[15]  Murali Krishna Ramanathan,et al.  Directed test generation to detect loop inefficiencies , 2016, SIGSOFT FSE.

[16]  Sven Apel,et al.  Distance-Based Sampling of Software Configuration Spaces , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[17]  Yuanyuan Zhou,et al.  Do not blame users for misconfigurations , 2013, SOSP.

[18]  Sergio Segura,et al.  Performance Metamorphic Testing: Motivation and Challenges , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER).

[19]  Isil Dillig,et al.  Static detection of asymptotic performance bugs in collection traversals , 2015, PLDI.

[20]  George C. Necula,et al.  EventBreak , 2014, OOPSLA.

[21]  Sven Apel,et al.  Using bad learners to find good configurations , 2017, ESEC/SIGSOFT FSE.

[22]  Xu Liu,et al.  Redundant Loads: A Software Inefficiency Indicator , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[23]  Shan Lu,et al.  Performance Diagnosis for Inefficient Loops , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[24]  Shan Lu,et al.  Statistical debugging for real-world performance problems , 2014, OOPSLA.

[25]  Guoqing Xu,et al.  Cachetor: detecting cacheable data to remove bloat , 2013, ESEC/FSE 2013.

[26]  Emery D. Berger,et al.  SHERIFF: precise detection and automatic mitigation of false sharing , 2011, OOPSLA '11.

[27]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[28]  Alvin Cheung,et al.  View-Centric Performance Optimization for Database-Backed Web Applications , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[29]  Thomas R. Gross,et al.  Performance regression testing of concurrent classes , 2014, ISSTA 2014.

[30]  Baowen Xu,et al.  Speedoo: Prioritizing Performance Optimization Opportunities , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[31]  Mohammed J. Zaki,et al.  Mining features for sequence classification , 1999, KDD '99.

[32]  Sergio Segura,et al.  Performance metamorphic testing: A Proof of concept , 2018, Inf. Softw. Technol..

[33]  Jean-Marc Jézéquel,et al.  Sampling Effect on Performance Prediction of Configurable Systems: A Case Study , 2020, ICPE.

[34]  Sven Apel,et al.  Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[35]  Norbert Siegmund,et al.  Transfer learning for performance modeling of configurable systems: An exploratory analysis , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[36]  Marcelo d'Amorim,et al.  Balancing Soundness and Efficiency for Practical Testing of Configurable Systems , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[37]  Thu D. Nguyen,et al.  Falling Off the Cliff: When Systems Go Nonlinear , 2005, HotOS.

[38]  Alvin Cheung,et al.  How not to Structure Your Database-Backed Web Applications: A Study of Performance Bugs in the Wild , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[39]  Ahmed E. Hassan,et al.  CacheOptimizer: helping developers configure caching frameworks for hibernate-based database-centric web applications , 2016, SIGSOFT FSE.

[40]  Tingting Yu,et al.  SyncProf: detecting, localizing, and optimizing synchronization bottlenecks , 2016, ISSTA.

[41]  Tingting Yu,et al.  Reproducing performance bug reports in server applications: The researchers' experiences , 2019, J. Syst. Softw..

[42]  Yang Liu,et al.  Generating Performance Distributions via Probabilistic Symbolic Execution , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[43]  Shan Lu,et al.  CARAMEL: Detecting and Fixing Performance Problems That Have Non-Intrusive Fixes , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[44]  Tsong Yueh Chen,et al.  Metamorphic Relations for Detection of Performance Anomalies , 2019, 2019 IEEE/ACM 4th International Workshop on Metamorphic Testing (MET).

[45]  Sven Apel,et al.  Performance-influence models for highly configurable systems , 2015, ESEC/SIGSOFT FSE.

[46]  Qi Luo,et al.  Automating performance bottleneck detection using search-based application profiling , 2015, ISSTA.