PerfBlower: Quickly Detecting Memory-Related Performance Problems via Amplification

Performance problems in managed languages are extremely difficult to find. Despite many efforts to find those problems, most existing work focuses on how to debug a user-provided test execution in which performance problems already manifest. It remains largely unknown how to effectively find performance bugs before software release. As a result, performance bugs often escape to production runs, hurting software reliability and user experience. This paper describes PerfBlower, a general performance testing framework that allows developers to quickly test Java programs to find memory-related performance problems. PerfBlower provides (1) a novel specification language ISL to describe a general class of performance problems that have observable symptoms; (2) an automated test oracle via \emph{virtual amplification}; and (3) precise reference-path-based diagnostic information via object mirroring. Using this framework, we have amplified three different types of problems. Our experimental results demonstrate that (1) ISL is expressive enough to describe various memory-related performance problems; (2) PerfBlower successfully distinguishes executions with and without problems; 8 unknown problems are quickly discovered under small workloads; and (3) PerfBlower outperforms existing detectors and does not miss any bugs studied before in the literature.

[1]  Dongmei Zhang,et al.  Performance debugging in the large via mining millions of stack traces , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[2]  Dongmei Zhang,et al.  Comprehending performance from real-world execution traces: a device-driver case , 2014, ASPLOS.

[3]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[4]  Amer Diwan,et al.  The DaCapo benchmarks: java benchmarking development and analysis , 2006, OOPSLA '06.

[5]  Matthew Arnold,et al.  Go with the flow: profiling copies to find runtime bloat , 2009, PLDI '09.

[6]  Bowen Zhou,et al.  WuKong: automatically detecting and localizing bugs that manifest at large system scales , 2013, HPDC '13.

[7]  Eran Yahav,et al.  Chameleon: adaptive selection of collections , 2009, PLDI '09.

[8]  Neil Immerman,et al.  What can the GC compute efficiently?: a language for heap assertions at GC time , 2010, SPLASH 2010.

[9]  Yannis Smaragdakis,et al.  DiSTiL: A Transformation Library for Data Structures , 1997, DSL.

[10]  Markus Mock,et al.  DyC: an expressive annotation-directed dynamic compiler for C , 2000, Theor. Comput. Sci..

[11]  Edith Schonberg,et al.  Finding low-utility data structures , 2010, PLDI '10.

[12]  Calvin Lin,et al.  An annotation language for optimizing software libraries , 1999, DSL '99.

[13]  Dawson R. Engler,et al.  Incorporating Application Semantics and Control into Compilation , 1997, DSL.

[14]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[15]  Christopher Stewart,et al.  EntomoModel: Understanding and Avoiding Performance Anomaly Manifestations , 2010, 2010 IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[16]  Shan Lu,et al.  Statistical debugging for real-world performance problems , 2014, OOPSLA.

[17]  Jeannette M. Wing.  Writing Larch interface language specifications , 1987, TOPL.

[18]  Mark T. Vandevoorde.  Exploiting specifications to improve program performance , 1993 .

[19]  Atanas Rountev,et al.  Detecting inefficiently-used containers to avoid bloat , 2010, PLDI '10.

[20]  Nick Mitchell,et al.  The causes of bloat, the limits of health , 2007, OOPSLA.

[21]  David R. O'Hallaron,et al.  Computer Systems: A Programmer's Perspective , 1991 .

[22]  Michael D. Bond,et al.  LeakChaser: helping programmers narrow down causes of memory leaks , 2011, PLDI '11.

[23]  Shan Lu,et al.  CARAMEL: Detecting and Fixing Performance Problems That Have Non-Intrusive Fixes , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[24]  Sorin Lerner,et al.  Verifying GPU kernels by test amplification , 2012, PLDI.

[25]  Dongmei Zhang,et al.  Context-sensitive delta inference for identifying workload-dependent performance bottlenecks , 2013, ISSTA.

[26]  Sebastian G. Elbaum,et al.  Amplifying tests to validate exception handling code , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[27]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[28]  Shan Lu,et al.  Toddler: Detecting performance problems via similar memory-access patterns , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[29]  Michael D. Bond,et al.  Probabilistic calling context , 2007, OOPSLA.

[30]  Michael D. Bond,et al.  Bell: bit-encoding online memory leak detection , 2006, ASPLOS XII.

[31]  Atanas Rountev,et al.  Precise memory leak detection for java software using container profiling , 2013, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[32]  Atanas Rountev,et al.  Static Detection of Loop-Invariant Data Structures , 2012, ECOOP.

[33]  Kathryn S. McKinley,et al.  Cork: dynamic memory leak detection for garbage-collected languages , 2007, POPL '07.

[34]  Michael D. Bond,et al.  Leak pruning , 2009, ASPLOS.

[35]  Isil Dillig,et al.  Static detection of asymptotic performance bugs in collection traversals , 2015, PLDI.

[36]  Nick Mitchell,et al.  Modeling Runtime Behavior in Framework-Based Applications , 2006, ECOOP.

[37]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.