Compiling a benchmark of documented multi-threaded bugs

Summary form only given. Testing multithreaded, concurrent, or distributed programs is acknowledged to be a very difficult task. We decided to create a benchmark of programs containing documented multithreaded bugs that can be used in the development of testing tool for the domain. In order to augment the benchmark with a sizable number of programs, we assigned students in a software testing class to write buggy multithreaded Java programs and document the bugs. This paper documents this experiment. We explain the task that was given to the students, go over the bugs that they put into the programs both intentionally and unintentionally, and show our findings. We believe this part of the benchmark shows typical programming practices, including bugs, of novice programmers. In grading the assignments, we used our technologies to look for undocumented bugs. In addition to finding many undocumented bugs, which was not surprising given that writing correct multithreaded code is difficult, we also found a number of bugs in our tools. We think this is a good indication of the expected utility of the benchmark for multithreaded testing tool creators.

[1]  Jong-Deok Choi,et al.  Efficient and precise datarace detection for multithreaded object-oriented programs , 2002, PLDI '02.

[2]  Yosi Ben-Asher,et al.  Heuristics for finding concurrent bugs , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[3]  Jong-Deok Choi,et al.  Deterministic replay of Java multithreaded applications , 1998, SPDT '98.

[4]  Eran Yahav,et al.  Verifying safety properties of concurrent Java programs using 3-valued logic , 2001, POPL '01.

[5]  Scott D. Stoller.  Model-Checking Multi-threaded Distributed Java Programs , 2000, SPIN.

[6]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[7]  A. Zeller.  Isolating cause-effect chains from computer programs , 2002, SIGSOFT '02/FSE-10.

[8]  Scott D. Stoller,et al.  Model-checking multi-threaded distributed Java programs , 2000, International Journal on Software Tools for Technology Transfer.

[9]  Scott D. Stoller,et al.  Run-Time Analysis for Atomicity , 2003, RV@CAV.

[10]  Stephen N. Freund,et al.  Atomizer: a dynamic atomicity checker for multithreaded programs , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[11]  Doug Lea.  Concurrent Programming in Java. Second Edition: Design Principles and Patterns , 1999 .

[12]  Daniel Hoffman,et al.  Tool Support for Testing Concurrent Java Components , 2003, IEEE Trans. Software Eng..

[13]  Jong-Deok Choi,et al.  Escape analysis for Java , 1999, OOPSLA '99.

[14]  Scott D. Stoller,et al.  Testing Concurrent Java Programs using Randomized Scheduling , 2002, RV@FLoC.

[15]  Klaus Havelund,et al.  Benchmark and framework for encouraging research on multi-threaded testing tools , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[16]  Alan Hartman,et al.  Using a model-based test generator to test for standard conformance , 2002, IBM Syst. J..

[17]  Watts S. Humphrey,et al.  Using A Defined and Measured Personal Software Process , 1996, IEEE Softw..

[18]  Eitan Farchi,et al.  Multithreaded Java program test generation , 2002, IBM Syst. J..

[19]  Cem Kaner,et al.  Bug Taxonomies: Use Them to Generate Better Tests 1 , 2003 .

[20]  Eitan Farchi,et al.  Concurrent bug patterns and how to test them , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[21]  Alan Bundy,et al.  Constructing Induction Rules for Deductive Synthesis Proofs , 2006, CLASE.