Testing concurrent programs to achieve high synchronization coverage

The effectiveness of software testing is often assessed by measuring coverage of some aspect of the software, such as its code. There is much research aimed at increasing code coverage of sequential software. However, there has been little research on increasing coverage for concurrent software. This paper presents a new technique that aims to achieve high coverage of concurrent programs by generating thread schedules to cover uncovered coverage requirements. Our technique first estimates synchronization-pair coverage requirements, and then generates thread schedules that are likely to cover uncovered coverage requirements. This paper also presents a description of a prototype tool that we implemented in Java, and the results of a set of studies we performed using the tool on a several open-source programs. The results show that, for our subject programs, our technique achieves higher coverage faster than random testing techniques; the estimation-based heuristic contributes substantially to the effectiveness of our technique.

[1]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[2]  Yuanyuan Zhou,et al.  CTrigger: exposing atomicity violation bugs from their hiding places , 2009, ASPLOS.

[3]  Lori L. Pollock,et al.  All-du-path coverage for parallel programs , 1998, ISSTA '98.

[4]  Eitan Farchi,et al.  Applications of synchronization coverage , 2005, PPoPP.

[5]  Tomás Vojnar,et al.  A platform for search-based testing of concurrent software , 2010, PADTAD '10.

[6]  Madan Musuvathi,et al.  Iterative context bounding for systematic testing of multithreaded programs , 2007, PLDI '07.

[7]  Eitan Farchi,et al.  Multithreaded Java program test generation , 2001, JGI '01.

[8]  Koushik Sen,et al.  CalFuzzer: An Extensible Active Testing Framework for Concurrent Programs , 2009, CAV.

[9]  Gregg Rothermel,et al.  A Scalable Distributed Concolic Testing Approach: An Empirical Evaluation , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[10]  Scott D. Stoller,et al.  Testing Concurrent Java Programs using Randomized Scheduling , 2002, RV@FLoC.

[11]  Chao Wang,et al.  Coverage guided systematic concurrency testing , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[12]  Koushik Sen,et al.  Race directed random testing of concurrent programs , 2008, PLDI '08.

[13]  Myra B. Cohen,et al.  Directed test suite augmentation: techniques and tradeoffs , 2010, FSE '10.

[14]  Akbar Siami Namin,et al.  The influence of size and coverage on test suite effectiveness , 2009, ISSTA.

[15]  Matthew B. Dwyer,et al.  Saturation-based testing of concurrent programs , 2009, ESEC/FSE '09.

[16]  Koushik Sen,et al.  A randomized dynamic program analysis technique for detecting real deadlocks , 2009, PLDI '09.

[17]  Tomás Vojnar,et al.  Coverage Metrics for Saturation-Based and Search-Based Testing of Concurrent Software , 2011, RV.

[18]  Eitan Farchi,et al.  Forcing small models of conditions on program interleaving for detection of concurrent bugs , 2009, PADTAD '09.

[19]  Shan Lu,et al.  A study of interleaving coverage criteria , 2007, ESEC-FSE companion '07.

[20]  Michael R. Lyu,et al.  The effect of code coverage on fault detection under different testing profiles , 2005, A-MOST.