CARAMEL: Detecting and Fixing Performance Problems That Have Non-Intrusive Fixes

Performance bugs are programming errors that slow down program execution. While existing techniques can detect various types of performance bugs, a crucial and practical aspect of performance bugs has not received the attention it deserves: how likely are developers to fix a performance bug? In practice, fixing a performance bug can have both benefits and drawbacks, and developers fix a performance bug only when the benefits outweigh the drawbacks. Unfortunately, for many performance bugs, the benefits and drawbacks are difficult to assess accurately. This paper presents CARAMEL, a novel static technique that detects and fixes performance bugs that have non-intrusive fixes likely to be adopted by developers. Each performance bug detected by CARAMEL is associated with a loop and a condition. When the condition becomes true during the loop execution, all the remaining computation performed by the loop is wasted. Developers typically fix such performance bugs because these bugs waste computation in loops and have non-intrusive fixes: when some condition becomes true dynamically, just break out of the loop. Given a program, CARAMEL detects such bugs statically and gives developers a potential source-level fix for each bug. We evaluate CARAMEL on real-world applications, including 11 popular Java applications (e.g., Groovy, Log4J, Lucene, Struts, Tomcat, etc) and 4 widely used C/C++ applications (Chromium, GCC, Mozilla, and My SQL). CARAMEL finds 61 new performance bugs in the Java applications and 89 new performance bugs in the C/C++ applications. Based on our bug reports, developers so far have fixed 51 and 65 performance bugs in the Java and C/C++ applications, respectively. Most of the remaining bugs are still under consideration by developers.

[1]  Westley Weimer,et al.  The road not taken: Estimating path execution frequency statically , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[2]  Qiang He,et al.  Automated analysis of performance and energy consumption for cloud applications , 2014, ICPE.

[3]  Chen Fu,et al.  Automatically finding performance problems with feedback-directed learning software testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[4]  Camil Demetrescu,et al.  Mining hot calling contexts in small space , 2011, PLDI '11.

[5]  Manish Gupta,et al.  Reuse, Recycle to De-bloat Software , 2011, ECOOP.

[6]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[7]  Thomas R. Gross,et al.  Performance regression testing of concurrent classes , 2014, ISSTA 2014.

[8]  Miryung Kim,et al.  Systematic editing: generating program transformations from an example , 2011, PLDI '11.

[9]  Claire Le Goues,et al.  A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[10]  Miryung Kim,et al.  Lase: Locating and applying systematic edits by learning from examples , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[11]  George C. Necula,et al.  EventBreak , 2014, OOPSLA.

[12]  Hoan Anh Nguyen,et al.  Recurring bug fixes in object-oriented programs , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[13]  Erik R. Altman,et al.  Performance analysis of idle programs , 2010, OOPSLA.

[14]  Koushik Sen,et al.  WISE: Automated test generation for worst-case complexity , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[15]  Claire Le Goues,et al.  Automatically finding patches using genetic programming , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[16]  Matthias Hauswirth,et al.  Evaluating the accuracy of Java profilers , 2010, PLDI '10.

[17]  Frank Tip,et al.  Automated repair of HTML generation errors in PHP applications using string constraint solving , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[18]  Edith Schonberg,et al.  Scalable Runtime Bloat Detection Using Abstract Dynamic Slicing , 2014, ACM Trans. Softw. Eng. Methodol..

[19]  Camil Demetrescu,et al.  Input-Sensitive Profiling , 2012, IEEE Transactions on Software Engineering.

[20]  Zhiqiang Ma,et al.  Ad Hoc Synchronization Considered Harmful , 2010, OSDI.

[21]  Ian Molyneaux.  The Art of Application Performance Testing - Help for Programmers and Quality Assurance , 2009 .

[22]  Dongmei Zhang,et al.  Context-sensitive delta inference for identifying workload-dependent performance bottlenecks , 2013, ISSTA.

[23]  Ranjit Jhala,et al.  Finding latent performance bugs in systems implementations , 2010, FSE '10.

[24]  Gang Yin,et al.  An online service-oriented performance profiling tool for cloud computing systems , 2013, Frontiers of Computer Science.

[25]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.

[26]  Sumit Gulwani,et al.  SPEED: precise and efficient static estimation of program computational complexity , 2009, POPL '09.

[27]  Martin Monperrus,et al.  Automatic repair of buggy if conditions and missing preconditions with SMT , 2014, CSTVA 2014.

[28]  Charles Zhang,et al.  Axis: Automatically fixing atomicity violations through solving control constraints , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[29]  Andreas Zeller,et al.  Generating Fixes from Object Behavior Anomalies , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[30]  Ahmed E. Hassan,et al.  Automatic detection of performance deviations in the load testing of Large Scale Systems , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[31]  Matthias Hauswirth,et al.  Algorithmic profiling , 2012, PLDI.

[32]  Steven P. Reiss,et al.  Using computer simulation to predict the performance of multithreaded programs , 2012, ICPE '12.

[33]  Xiao Ma,et al.  Performance regression testing target prioritization via performance risk analysis , 2014, ICSE.

[34]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[35]  Lars Grunske,et al.  Model-based performance analysis of software architectures under uncertainty , 2013, QoSA '13.

[36]  Jaechang Nam,et al.  Automatic patch generation learned from human-written patches , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[37]  Gunter Saake,et al.  Predicting performance via automated feature-interaction detection , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[38]  Alexander Aiken,et al.  Effective static race detection for Java , 2006, PLDI '06.

[39]  Miryung Kim,et al.  Recommending Program Transformations - Automating Repetitive Software Changes , 2014, Recommendation Systems in Software Engineering.

[40]  David R. O'Hallaron,et al.  Computer Systems: A Programmer's Perspective , 1991 .

[41]  Dongmei Zhang,et al.  Performance debugging in the large via mining millions of stack traces , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[42]  Dongmei Zhang,et al.  Comprehending performance from real-world execution traces: a device-driver case , 2014, ASPLOS.

[43]  Matthew B. Dwyer,et al.  Automatic generation of load tests , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[44]  Sven Apel,et al.  Variability-aware performance prediction: A statistical learning approach , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[45]  Dawei Qi,et al.  SemFix: Program repair via semantic analysis , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[46]  Tevfik Bultan,et al.  Semantic differential repair for input validation and sanitization , 2014, ISSTA 2014.

[47]  Shan Lu,et al.  Toddler: Detecting performance problems via similar memory-access patterns , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[48]  Claire Le Goues,et al.  Current challenges in automatic software repair , 2013, Software Quality Journal.

[49]  Simon Goldsmith,et al.  Measuring empirical computational complexity , 2007, ESEC-FSE '07.

[50]  Emery D. Berger,et al.  SHERIFF: precise detection and automatic mitigation of false sharing , 2011, OOPSLA '11.

[51]  Barbara G. Ryder,et al.  A scalable technique for characterizing the usage of temporaries in framework-intensive Java applications , 2008, SIGSOFT '08/FSE-16.

[52]  Guru Venkataramani,et al.  LIME: a framework for debugging load imbalance in multi-threaded execution , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[53]  Shan Lu,et al.  Statistical debugging for real-world performance problems , 2014, OOPSLA.

[54]  Guoqing Xu,et al.  Cachetor: detecting cacheable data to remove bloat , 2013, ESEC/FSE 2013.