Advances in noise‐based testing of concurrent software

Testing of concurrent software written in programming languages like Java and C/C++ is a highly challenging task owing to the many possible interactions among threads. A simple, cheap, and effective approach that addresses this challenge is testing with noise injection, which influences the scheduling so that different interleavings of concurrent actions are witnessed. In this paper, multiple results achieved recently in the area of noise‐injection‐based testing by the authors are presented in a unified and extended way. In particular, various concurrency coverage metrics are presented first. Then, multiple heuristics for solving the noise placement problem (i.e. where and when to generate noise) as well as the noise seeding problem (i.e. how to generate the noise) are introduced and experimentally evaluated. In addition, several new heuristics are proposed and included into the evaluation too. Recommendations on how to set up noise‐based testing for particular scenarios are then given. Finally, a novel use of the genetic algorithm for finding suitable combinations of the many parameters of tests and noise techniques is presented. Copyright © 2014 John Wiley & Sons, Ltd.

[1]  Shin Hong,et al.  Testing concurrent programs to achieve high synchronization coverage , 2012, ISSTA 2012.

[2]  Yosi Ben-Asher,et al.  Heuristics for finding concurrent bugs , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[3]  Koushik Sen,et al.  A randomized dynamic program analysis technique for detecting real deadlocks , 2009, PLDI '09.

[4]  Enrique Alba,et al.  Comparing Metaheuristic Algorithms for Error Detection in Java Programs , 2011, SSBSE.

[5]  Timo Latvala,et al.  Explaining Intermittent Concurrent Bugs by Minimizing Scheduling Noise , 2006, Haifa Verification Conference.

[6]  Tomás Vojnar,et al.  AtomRace: data race and atomicity violation detector and healer , 2008, PADTAD '08.

[7]  Elad Yom-Tov,et al.  Instrumenting where it hurts: an automatic concurrent debugging technique , 2007, ISSTA '07.

[8]  Tomás Vojnar,et al.  DA-BMC: A Tool Chain Combining Dynamic Analysis and Bounded Model Checking , 2011, RV.

[9]  Yuanyuan Zhou,et al.  AVIO: Detecting Atomicity Violations via Access-Interleaving Invariants , 2007, IEEE Micro.

[10]  Gordon Fraser,et al.  Generating Unit Tests for Concurrent Classes , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[11]  Scott D. Stoller,et al.  Testing Concurrent Java Programs using Randomized Scheduling , 2002, RV@FLoC.

[12]  Enrique Alba,et al.  Finding deadlocks in large concurrent java programs using genetic algorithms , 2008, GECCO '08.

[13]  Armin Biere,et al.  Applying static analysis to large-scale, multi-threaded Java programs , 2001, Proceedings 2001 Australian Software Engineering Conference.

[14]  Tomás Vojnar,et al.  A Concurrency Testing Tool and Its Plug-Ins for Dynamic Analysis and Runtime Healing , 2009, RV.

[15]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[16]  Tomás Vojnar,et al.  Noise-based testing and analysis of multi-threaded C/C++ programs on the binary level , 2012, PADTAD 2012.

[17]  Tomás Vojnar,et al.  ANaConDA: A Framework for Analysing Multi-threaded C/C++ Programs on the Binary Level , 2012, RV.

[18]  David Gay,et al.  Effective static deadlock detection , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[19]  Yosi Ben-Asher,et al.  Noise Makers Need to Know Where to be Silent – Producing Schedules That Find Bugs , 2006, Second International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (isola 2006).

[20]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multi-threaded programs , 1997, TOCS.

[21]  Tomás Vojnar,et al.  Healing data races on-the-fly , 2007, PADTAD '07.

[22]  Sarfraz Khurshid,et al.  Exploring very large state spaces using genetic algorithms , 2004, International Journal on Software Tools for Technology Transfer.

[23]  David R. White.  Software review: the ECJ toolkit , 2011, Genetic Programming and Evolvable Machines.

[24]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[25]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[26]  Matthew B. Dwyer,et al.  Saturation-based testing of concurrent programs , 2009, ESEC/FSE '09.

[27]  Richard N. Taylor,et al.  Structural Testing of Concurrent Programs , 1992, IEEE Trans. Software Eng..

[28]  Jeremy Manson,et al.  The Java memory model , 2005, POPL '05.

[29]  Shaz Qadeer,et al.  CHESS: A Systematic Testing Tool for Concurrent Software , 2007 .

[30]  Thomas R. Gross,et al.  Object race detection , 2001, OOPSLA '01.

[31]  B. Maddock,et al.  FROM DESIGN TO IMPLEMENTATION , 1982 .

[32]  Harish Patil,et al.  Pin: building customized program analysis tools with dynamic instrumentation , 2005, PLDI '05.

[33]  Lori L. Pollock,et al.  All-du-path coverage for parallel programs , 1998, ISSTA '98.

[34]  Eitan Farchi,et al.  Forcing small models of conditions on program interleaving for detection of concurrent bugs , 2009, PADTAD '09.

[35]  Satish Narayanasamy,et al.  Maple: a coverage-driven testing tool for multithreaded programs , 2012, OOPSLA '12.

[36]  Shan Lu,et al.  A study of interleaving coverage criteria , 2007, ESEC-FSE companion '07.

[37]  Miguel Isabel.  Testing of Concurrent Programs , 2016, ICLP.

[38]  Tomás Vojnar,et al.  Coverage Metrics for Saturation-Based and Search-Based Testing of Concurrent Software , 2011, RV.

[39]  Yaniv Eytani.  Concurrent Java Test Generation as a Search Problem , 2006, Electron. Notes Theor. Comput. Sci..

[40]  Tomás Vojnar,et al.  Noise Injection Heuristics for Concurrency Testing , 2011, MEMICS.

[41]  José Manuel Cantera,et al.  Delivering Mobile Enterprise Services on Morfeo’s MC Open Source Platform , 2006, 7th International Conference on Mobile Data Management (MDM'06).

[42]  David Hovemeyer,et al.  Finding Concurrency Bugs in Java , 2004 .

[43]  Jeremy S. Bradbury,et al.  How Good is Static Analysis at Finding Concurrency Bugs? , 2010, 2010 10th IEEE Working Conference on Source Code Analysis and Manipulation.

[44]  Eitan Farchi,et al.  Multithreaded Java program test generation , 2001, JGI '01.

[45]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[46]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[47]  Assaf Schuster,et al.  Efficient on-the-fly data race detection in multithreaded C++ programs , 2003, PPoPP '03.

[48]  Assaf Schuster,et al.  Efficient on-the-fly data race detection in multithreaded C++ programs , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[49]  Enrique Alba,et al.  Searching for liveness property violations in concurrent systems with ACO , 2008, GECCO '08.

[50]  Christel Baier,et al.  Principles of Model Checking (Representation and Mind Series) , 2008 .

[51]  Zdenek Letko.  Analysis and Testing of Concurrent Programs , 2013 .

[52]  Christel Baier,et al.  Principles of model checking , 2008 .

[53]  Klaus Havelund,et al.  Dynamic Deadlock Analysis of Multi-threaded Programs , 2005, Haifa Verification Conference.

[54]  Koushik Sen,et al.  Randomized active atomicity violation detection in concurrent programs , 2008, SIGSOFT '08/FSE-16.

[55]  Koushik Sen,et al.  Race directed random testing of concurrent programs , 2008, PLDI '08.

[56]  Junfeng Yang,et al.  Sound and precise analysis of parallel programs through schedule specialization , 2012, PLDI.

[57]  Enrique Alba,et al.  Finding safety errors with ACO , 2007, GECCO '07.

[58]  Eitan Farchi,et al.  Framework for testing multi‐threaded Java programs , 2003, Concurr. Comput. Pract. Exp..

[59]  Eitan Farchi,et al.  Applications of synchronization coverage , 2005, PPoPP.

[60]  Tomás Vojnar,et al.  A platform for search-based testing of concurrent software , 2010, PADTAD '10.

[61]  Stephan Merz,et al.  Model Checking , 2000 .

[62]  David Holmes,et al.  Java Concurrency in Practice , 2006 .

[63]  Serdar Tasiran,et al.  Goldilocks: a race and transaction-aware java runtime , 2007, PLDI '07.

[64]  N TaylorRichard,et al.  Structural Testing of Concurrent Programs , 1992 .

[65]  Tomás Vojnar,et al.  Testing of Concurrent Programs Using Genetic Algorithms , 2012, SSBSE.

[66]  C. Pasareanu,et al.  Lifecycle Verification of the NASA Ames K 9 Rover Executive , 2004 .