Demonic Testing of Concurrent Programs

Testing presents a daunting challenge for concurrent programs, as non-deterministic scheduling defeats reproducibility. The problem is even harder if, rather than testing entire systems, one tries to test individual components, for example to assess them for thread-safety. We present demonic testing, a technique combining the tangible results of unit testing with the rigour of formal rely-guarantee reasoning to provide deterministic unit testing for concurrent programs. Deterministic execution is provided by abstracting threads away via rely-guarantee reasoning, and replacing them with "demonic" sequences of interfering instructions that drive the program to break invariants. Demonic testing reuses existing unit tests to drive the routine under test, using the execution to discover demonic interference. Programs carry contract-based rely-guarantee style specifications to express what sort of thread interference should be tolerated. Aiding the demonic testing technique is an interference synthesis tool we have implemented based on SMT solving. The technique is shown to find errors in contracted versions of several benchmark applications.

[1]  Chao Wang,et al.  Coverage guided systematic concurrency testing , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[2]  Eitan Farchi,et al.  Framework for testing multi‐threaded Java programs , 2003, Concurr. Comput. Pract. Exp..

[3]  Satish Narayanasamy,et al.  A case for an interleaving constrained shared-memory multi-processor , 2009, ISCA '09.

[4]  Cliff B. Jones,et al.  Developing methods for computer programs including a notion of interference , 1981 .

[5]  Yuanyuan Zhou,et al.  CTrigger: exposing atomicity violation bugs from their hiding places , 2009, ASPLOS.

[6]  Bor-Yuh Evan Chang,et al.  Boogie: A Modular Reusable Verifier for Object-Oriented Programs , 2005, FMCO.

[7]  Chao Wang,et al.  Contessa: Concurrency Testing Augmented with Symbolic Analysis , 2010, CAV.

[8]  Craig A. Knoblock,et al.  PDDL-the planning domain definition language , 1998 .

[9]  Eran Yahav,et al.  Testing atomicity of composed concurrent operations , 2011, OOPSLA '11.

[10]  Jong-Deok Choi,et al.  Deterministic replay of Java multithreaded applications , 1998, SPDT '98.

[11]  Albert L. Baker,et al.  Preliminary design of JML: a behavioral interface specification language for java , 2006, SOEN.

[12]  Bertrand Meyer,et al.  Programs That Test Themselves , 2009, Computer.

[13]  Bertrand Meyer,et al.  Object-Oriented Software Construction, 2nd Edition , 1997 .

[14]  簡聰富,et al.  物件導向軟體之架構(Object-Oriented Software Construction)探討 , 1989 .

[15]  C. Pasareanu,et al.  Assume-guarantee testing , 2006, SAVCBS '05.

[16]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[17]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[18]  Satish Narayanasamy,et al.  Automatically classifying benign and harmful data races using replay analysis , 2007, PLDI '07.

[19]  Lionel C. Briand,et al.  On the Effectiveness of Contracts as Test Oracles in the Detection and Diagnosis of Race Conditions and Deadlocks in Concurrent Object-Oriented Software , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[20]  Koushik Sen,et al.  Race directed random testing of concurrent programs , 2008, PLDI '08.

[21]  Jürgen Dingel,et al.  Computer-assisted assume/guarantee reasoning with VeriSoft , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[22]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[23]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.