Multi-objective Genetic Optimization for Noise-Based Testing of Concurrent Software

Testing of multi-threaded programs is a demanding work due to the many possible thread interleavings one should examine. The noise injection technique helps to increase the number of thread interleavings examined during repeated test executions provided that a suitable setting of noise injection heuristics is used. The problem of finding such a setting, i.e., the so called test and noise configuration search problem (TNCS problem), is not easy to solve. In this paper, we show how to apply a multi-objective genetic algorithm (MOGA) to the TNCS problem. In particular, we focus on generation of TNCS solutions that cover a high number of distinct interleavings (especially those which are rare) and provide stable results at the same time. To achieve this goal, we study suitable metrics and ways how to suppress effects of non-deterministic thread scheduling on the proposed MOGA-based approach. We also discuss a choice of a concrete MOGA and its parameters suitable for our setting. Finally, we show on a set of benchmark programs that our approach provides better results when compared to the commonly used random approach as well as to the sooner proposed use of a single-objective genetic approach.

[1]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[2]  J. David Morgenthaler,et al.  Using FindBugs on production software , 2007, OOPSLA '07.

[3]  Kalyanmoy Deb,et al.  Solving Bilevel Multi-Objective Optimization Problems Using Evolutionary Algorithms , 2009, EMO.

[4]  Tomás Vojnar,et al.  Coverage Metrics for Saturation-Based and Search-Based Testing of Concurrent Software , 2011, RV.

[5]  Satish Narayanasamy,et al.  Maple: a coverage-driven testing tool for multithreaded programs , 2012, OOPSLA '12.

[6]  Eitan Farchi,et al.  Cross-Entropy Based Testing , 2007, Formal Methods in Computer Aided Design (FMCAD'07).

[7]  Yuanyuan Zhou,et al.  AVIO: Detecting Atomicity Violations via Access-Interleaving Invariants , 2007, IEEE Micro.

[8]  Shin Hong,et al.  Testing concurrent programs to achieve high synchronization coverage , 2012, ISSTA 2012.

[9]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[10]  Gordon Fraser,et al.  Generating Unit Tests for Concurrent Classes , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[11]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multi-threaded programs , 1997, TOCS.

[12]  Tomás Vojnar,et al.  Testing of Concurrent Programs Using Genetic Algorithms , 2012, SSBSE.

[13]  Stephan Merz,et al.  Model Checking , 2000 .

[14]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[15]  David Holmes,et al.  Java Concurrency in Practice , 2006 .

[16]  L.A. Smith,et al.  A Parallel Java Grande Benchmark Suite , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[17]  Eckart Zitzler,et al.  Evolutionary algorithms for multiobjective optimization: methods and applications , 1999 .

[18]  Serdar Tasiran,et al.  Goldilocks: a race and transaction-aware java runtime , 2007, PLDI '07.

[19]  Jürgen Branke,et al.  Evolutionary optimization in uncertain environments-a survey , 2005, IEEE Transactions on Evolutionary Computation.

[20]  Klaus Havelund,et al.  Dynamic Deadlock Analysis of Multi-threaded Programs , 2005, Haifa Verification Conference.

[21]  Enrique Alba,et al.  Finding safety errors with ACO , 2007, GECCO '07.

[22]  Eitan Farchi,et al.  Framework for testing multi‐threaded Java programs , 2003, Concurr. Comput. Pract. Exp..

[23]  Eitan Farchi,et al.  Applications of synchronization coverage , 2005, PPoPP.

[24]  Tomás Vojnar,et al.  A platform for search-based testing of concurrent software , 2010, PADTAD '10.

[25]  David R. White Software review: the ECJ toolkit , 2011, Genetic Programming and Evolvable Machines.

[26]  John A. Clark,et al.  Searching for Safety Violations Using Estimation of Distribution Algorithms , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[27]  R. K. Ursem Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[28]  Tomás Vojnar,et al.  Noise Injection Heuristics for Concurrency Testing , 2011, MEMICS.

[29]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[30]  Tomás Vojnar,et al.  A Concurrency Testing Tool and Its Plug-Ins for Dynamic Analysis and Runtime Healing , 2009, RV.

[31]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[32]  Julian F. Miller,et al.  Genetic and Evolutionary Computation — GECCO 2003 , 2003, Lecture Notes in Computer Science.

[34]  Hisao Ishibuchi,et al.  A Similarity-Based Mating Scheme for Evolutionary Multiobjective Optimization , 2003, GECCO.

[35]  Petr Hliněný,et al.  Mathematical and Engineering Methods in Computer Science , 2012, Lecture Notes in Computer Science.

[36]  Tomás Vojnar,et al.  Healing data races on-the-fly , 2007, PADTAD '07.

[37]  Sarfraz Khurshid,et al.  Exploring very large state spaces using genetic algorithms , 2004, International Journal on Software Tools for Technology Transfer.