Learning from Source Code History to Identify Performance Failures

Source code changes may inadvertently introduce performance regressions. Benchmarking each software version is traditionally employed to identify performance regressions. Although effective, this exhaustive approach is hard to carry out in practice. This paper contrasts source code changes against performance variations. By analyzing 1,288 software versions from 17 open source projects, we identified 10 source code changes leading to a performance variation (improvement or regression). We have produced a cost model to infer whether a software commit introduces a performance variation by analyzing the source code and sampling the execution of a few versions. By profiling the execution of only 17% of the versions, our model is able to identify 83% of the performance regressions greater than 5% and 100% of the regressions greater than 50%.

[1]  Hong Mei,et al.  An experimental comparison of four test suite reduction techniques , 2006, ICSE.

[2]  Johan A. Pouwelse,et al.  Detecting and analyzing I/O performance regressions , 2014, J. Softw. Evol. Process..

[3]  Ahmed E. Hassan,et al.  Automated detection of performance regressions using statistical process control techniques , 2012, ICPE '12.

[4]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[5]  Gilbert Hamann,et al.  Automatic Comparison of Load Tests to Support the Performance Analysis of Large Enterprise Systems , 2010, 2010 14th European Conference on Software Maintenance and Reengineering.

[6]  Shan Lu,et al.  Toddler: Detecting performance problems via similar memory-access patterns , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[7]  Roozbeh Farahbod,et al.  Automated root cause isolation of performance regressions during software development , 2013, ICPE '13.

[8]  Ahmed E. Hassan,et al.  An industrial case study of automatically identifying performance regression-causes , 2014, MSR 2014.

[9]  Alexandre Bergel,et al.  Tracking down performance variation against source code evolution , 2015, DLS.

[10]  Ahmed E. Hassan,et al.  Automated Detection of Performance Regressions Using Regression Models on Clustered Performance Counters , 2015, ICPE.

[11]  Alexandre Bergel,et al.  Counting Messages as a Proxy for Average Execution Time in Pharo , 2011, ECOOP.

[12]  Xiao Ma,et al.  Performance regression testing target prioritization via performance risk analysis , 2014, ICSE.

[13]  Meir M. Lehman,et al.  A Model of Large Program Development , 1976, IBM Syst. J..

[14]  Ian Molyneaux.  The Art of Application Performance Testing - Help for Programmers and Quality Assurance , 2009 .

[15]  Gregg Rothermel,et al.  Test case prioritization: an empirical study , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[16]  Ahmed E. Hassan,et al.  A qualitative study on performance bugs , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[17]  Liming Zhu,et al.  DevOps - A Software Architect's Perspective , 2015, SEI series in software engineering.

[18]  Claes Wohlin,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[19]  John C. Grundy,et al.  Performance Analysis for Object-Oriented Software: A Systematic Mapping , 2015, IEEE Transactions on Software Engineering.

[20]  Jakob Engblom,et al.  The worst-case execution-time problem—overview of methods and survey of tools , 2008, TECS.

[21]  Qi Luo,et al.  Automating performance bottleneck detection using search-based application profiling , 2015, ISSTA.

[22]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[23]  Andrew Glover,et al.  Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series) , 2007 .

[24]  Sofware Evolution And....  Software evolution and feedback : theory and practice , 2014 .

[25]  Dewayne E. Perry,et al.  Metrics and laws of software evolution-the nineties view , 1997, Proceedings Fourth International Software Metrics Symposium.

[26]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[27]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[28]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[29]  Tian Jiang,et al.  Discovering, reporting, and fixing performance bugs , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[30]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[31]  Dongmei Zhang,et al.  Performance debugging in the large via mining millions of stack traces , 2012, 2012 34th International Conference on Software Engineering (ICSE).