Toward a Benchmark for Multi-Threaded Testing Tools

Looking for intermittent bugs is a problem that has been getting prominence in testing. Multi-threaded code is becoming very common, mostly on the server side. As there is no silver bullet solution, research focuses on a variety of partial solutions. We outline a road map for combining the research on the different disciplines of testing multi-threaded programs and on evaluating its quality. The project goals are to create a benchmark that can be used to evaluate different solutions, to create a framework with open API's that enables combining techniques in the multithreading domain, and to create a focus for the research in this area around which a community of people who try to solve similar problems with different techniques, could congregate. The benchmark, apart from containing programs with documented bugs, includes other artifacts, such as traces, that are used for evaluating some of the technologies. We have started creating such a bench mrk and detail the lesson learned in the process. The framework will enable technology developers, for example, race detectors, to concentrate on their components and use other ready made components, (e.g., instrumentor) to create a testing solution.

[1]  Stephen N. Freund,et al.  Atomizer: a dynamic atomicity checker for multithreaded programs , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[2]  Péter Kacsuk.  Parallel program development and execution in the grid , 2002, Proceedings. International Conference on Parallel Computing in Electrical Engineering.

[3]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multi-threaded programs , 1997, TOCS.

[4]  Yosi Ben-Asher,et al.  Heuristics for finding concurrent bugs , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[5]  Jong-Deok Choi,et al.  Efficient and precise datarace detection for multithreaded object-oriented programs , 2002, PLDI '02.

[6]  José C. Cunha,et al.  Parallel program development for cluster computing: methodology, tools and integrated environments , 2001 .

[7]  James A. Whittaker.  How to Break Software , 2000 .

[8]  Stefan Hanenberg,et al.  Implementing Known Concepts in AspectJ , 2003 .

[9]  Thomas A. Henzinger,et al.  Lazy abstraction , 2002, POPL '02.

[10]  Eitan Farchi,et al.  Multithreaded Java program test generation , 2002, IBM Syst. J..

[11]  Martin C. Rinard,et al.  ACM Conference on Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), November 2002 Ownership Types for Safe Programming: Preventing Data Races and Deadlocks , 2022 .

[12]  Cem Kaner,et al.  Bug Taxonomies: Use Them to Generate Better Tests 1 , 2003 .

[13]  Shmuel Ur.  Testing Multi-threaded Java Programs , 2002 .

[14]  Eitan Farchi,et al.  Concurrent bug patterns and how to test them , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[15]  Dan Grossman,et al.  Type-safe multithreading in cyclone , 2003, TLDI '03.

[16]  Dawson R. Engler,et al.  A system and language for building system-specific, static analyses , 2002, PLDI '02.

[17]  Jong-Deok Choi,et al.  Deterministic replay of Java multithreaded applications , 1998, SPDT '98.

[18]  Eran Yahav,et al.  Automatically Verifying Concurrent Queue Algorithms , 2003, SoftMC@CAV.

[19]  Jong-Deok Choi,et al.  Hybrid dynamic data race detection , 2003, PPoPP '03.

[20]  D. Engler,et al.  CMC: a pragmatic approach to model checking real code , 2002, OSDI '02.

[21]  Thomas W. Reps,et al.  Putting static analysis to work for verification: A case study , 2000, ISSTA '00.

[22]  Eran Yahav,et al.  Verifying safety properties of concurrent Java programs using 3-valued logic , 2001, POPL '01.

[23]  Thomas R. Gross,et al.  Static conflict analysis for multi-threaded object-oriented programs , 2003, PLDI '03.

[24]  Dawson R. Engler,et al.  RacerX: effective, static detection of race conditions and deadlocks , 2003, SOSP '03.

[25]  Sorin Lerner,et al.  ESP: path-sensitive program verification in polynomial time , 2002, PLDI '02.

[26]  Assaf Schuster,et al.  Toward Integration of Data Race Detection in DSM Systems , 1999, J. Parallel Distributed Comput..

[27]  Alan Hartman,et al.  A test execution environment running abstract tests for distributed software proceedings of sea 2002 , 2002 .

[28]  Scott D. Stoller,et al.  Testing Concurrent Java Programs using Randomized Scheduling , 2002, RV@FLoC.

[29]  Stephen Asbury,et al.  Developing Java Enterprise Applications , 1999 .

[30]  Klaus Havelund,et al.  Model checking JAVA programs using JAVA PathFinder , 2000, International Journal on Software Tools for Technology Transfer.

[31]  Grigore Rosu,et al.  Monitoring Java Programs with Java PathExplorer , 2001, RV@CAV.

[32]  George S. Avrunin,et al.  Comparing Finite-State Verification Techniques for Concurrent Software , 1999 .

[33]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[34]  Matthew B. Dwyer,et al.  Bandera: extracting finite-state models from Java source code , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[35]  Jerry J. Harrow.  Runtime Checking of Multithreaded Applications with Visual Threads , 2000, SPIN.

[36]  James R. Larus,et al.  Protocol-based data-race detection , 1998, SPDT '98.

[37]  Alan Bundy,et al.  Constructing Induction Rules for Deductive Synthesis Proofs , 2006, CLASE.

[38]  Lori L. Pollock,et al.  All-du-path coverage for parallel programs , 1998, ISSTA '98.

[39]  Jong-Deok Choi,et al.  Escape analysis for Java , 1999, OOPSLA '99.

[40]  Stephen N. Freund,et al.  Detecting race conditions in large programs , 2001, PASTE '01.

[41]  Barton P. Miller,et al.  Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services , 1995 .

[42]  Scott D. Stoller,et al.  Model-checking multi-threaded distributed Java programs , 2000, International Journal on Software Tools for Technology Transfer.

[43]  Scott D. Stoller,et al.  Run-Time Analysis for Atomicity , 2003, RV@CAV.

[44]  Barton P. Miller,et al.  Detecting Data Races in Parallel Program Executions , 1989 .

[45]  Deepak Goyal,et al.  Deriving specialized program analyses for certifying component-client conformance , 2002, PLDI '02.

[46]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[47]  Gerard J. Holzmann,et al.  SOFTWARE TESTING, VERIFICATION AND RELIABILITY , 2022 .

[48]  Daniel Hoffman,et al.  Tool Support for Testing Concurrent Java Components , 2003, IEEE Trans. Software Eng..

[49]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .