Maple: a coverage-driven testing tool for multithreaded programs

Testing multithreaded programs is a hard problem, because it is challenging to expose those rare interleavings that can trigger a concurrency bug. We propose a new thread interleaving coverage-driven testing tool called Maple that seeks to expose untested thread interleavings as much as possible. It memoizes tested interleavings and actively seeks to expose untested interleavings for a given test input to increase interleaving coverage. We discuss several solutions to realize the above goal. First, we discuss a coverage metric based on a set of interleaving idioms. Second, we discuss an online technique to predict untested interleavings that can potentially be exposed for a given test input. Finally, the predicted untested interleavings are exposed by actively controlling the thread schedule while executing for the test input. We discuss our experiences in using the tool to expose several known and unknown bugs in real-world applications such as Apache and MySQL.

[1]  Yuanyuan Zhou,et al.  CTrigger: exposing atomicity violation bugs from their hiding places , 2009, ASPLOS.

[2]  Dawson R. Engler,et al.  RacerX: effective, static detection of race conditions and deadlocks , 2003, SOSP '03.

[3]  Anoop Gupta,et al.  The SPLASH-2 programs: characterization and methodological considerations , 1995, ISCA.

[4]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[5]  Harish Patil,et al.  Pin: building customized program analysis tools with dynamic instrumentation , 2005, PLDI '05.

[6]  Cormac Flanagan,et al.  A type and effect system for atomicity , 2003, PLDI.

[7]  Stephen N. Freund,et al.  FastTrack: efficient and precise dynamic race detection , 2009, PLDI '09.

[8]  Eitan Farchi,et al.  Applications of synchronization coverage , 2005, PPoPP.

[9]  Jeff Huang,et al.  Persuasive prediction of concurrency access anomalies , 2011, ISSTA '11.

[10]  Satish Narayanasamy,et al.  A case for an interleaving constrained shared-memory multi-processor , 2009, ISCA '09.

[11]  Koushik Sen,et al.  Randomized active atomicity violation detection in concurrent programs , 2008, SIGSOFT '08/FSE-16.

[12]  Koushik Sen,et al.  Race directed random testing of concurrent programs , 2008, PLDI '08.

[13]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[14]  Chao Wang,et al.  Coverage guided systematic concurrency testing , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[15]  Daniel Jackson,et al.  Elements of style: analyzing a software design feature with a counterexample detector , 1996, ISSTA '96.

[16]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[17]  Stephen N. Freund,et al.  Velodrome: a sound and complete dynamic atomicity checker for multithreaded programs , 2008, PLDI '08.

[18]  Grigore Rosu,et al.  Improved multithreaded unit testing , 2011, ESEC/FSE '11.

[19]  Frank Tip,et al.  Associating synchronization constraints with data in an object-oriented language , 2006, POPL '06.

[20]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[21]  Francesco Sorrentino,et al.  PENELOPE: weaving threads to expose atomicity violations , 2010, FSE '10.

[22]  Shing-Chi Cheung,et al.  Detecting atomic-set serializability violations in multithreaded programs through active randomized testing , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[23]  Madan Musuvathi,et al.  Fair stateless model checking , 2008, PLDI '08.

[24]  Shaz Qadeer,et al.  Partial-Order Reduction for Context-Bounded State Exploration , 2007 .

[25]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[26]  Eitan Farchi,et al.  Multithreaded Java program test generation , 2002, IBM Syst. J..

[27]  Shan Lu,et al.  ConSeq: detecting concurrency bugs through sequential errors , 2011, ASPLOS XVI.

[28]  Chao Wang,et al.  Universal Causality Graphs: A Precise Happens-Before Model for Detecting Bugs in Concurrent Programs , 2010, CAV.

[29]  Matthew B. Dwyer,et al.  Saturation-based testing of concurrent programs , 2009, ESEC/FSE '09.

[30]  Lori L. Pollock,et al.  All-du-path coverage for parallel programs , 1998, ISSTA '98.

[31]  Brandon Lucia,et al.  Atom-Aid: Detecting and Surviving Atomicity Violations , 2009, IEEE Micro.

[32]  Xiangyu Zhang,et al.  Analyzing multicore dumps to facilitate concurrency bug reproduction , 2010, ASPLOS XV.

[33]  Shan Lu,et al.  A study of interleaving coverage criteria , 2007, ESEC-FSE '07.

[34]  Sebastian Burckhardt,et al.  Effective ? , 2010 .

[35]  Richard N. Taylor,et al.  Structural Testing of Concurrent Programs , 1992, IEEE Trans. Software Eng..

[36]  Patrice Godefroid,et al.  Partial-Order Methods for the Verification of Concurrent Systems , 1996, Lecture Notes in Computer Science.

[37]  Pravesh Kothari,et al.  A randomized scheduler with probabilistic guarantees of finding bugs , 2010, ASPLOS XV.

[38]  Tomás Vojnar,et al.  Coverage Metrics for Saturation-Based and Search-Based Testing of Concurrent Software , 2011, RV.

[39]  Shan Lu,et al.  ConMem: detecting severe concurrency bugs through an effect-oriented approach , 2010, ASPLOS XV.

[40]  Xiao Ma,et al.  MUVI: automatically inferring multi-variable access correlations and detecting related semantic and concurrency bugs , 2007, SOSP.

[41]  Sorin Lerner,et al.  RELAY: static race detection on millions of lines of code , 2007, ESEC-FSE '07.

[42]  David Gay,et al.  Effective static deadlock detection , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[43]  Paul Barford,et al.  Generating representative Web workloads for network and server performance evaluation , 1998, SIGMETRICS '98/PERFORMANCE '98.

[44]  Yuanyuan Zhou,et al.  AVIO: Detecting Atomicity Violations via Access-Interleaving Invariants , 2007, IEEE Micro.

[45]  Koushik Sen,et al.  A randomized dynamic program analysis technique for detecting real deadlocks , 2009, PLDI '09.

[46]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[47]  James Cownie,et al.  PinPlay: a framework for deterministic replay and reproducible analysis of parallel programs , 2010, CGO '10.

[48]  Patrice Godefroid,et al.  Dynamic partial-order reduction for model checking software , 2005, POPL '05.

[49]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.