Efficient mutation testing of multithreaded code

Mutation testing is a well‐established method for measuring and improving the quality of test suites. A major cost of mutation testing is the time required to execute the test suite on all the mutants. This cost is even greater when the system under test is multithreaded: not only are test cases from the test suite executed on many mutants but also each test case is executed—or more precisely, explored—for multiple possible thread schedules. This paper introduces a general framework for efficient exploration that can reduce the time for mutation testing of multithreaded code. The paper presents five techniques (four optimizations and one heuristic) that are implemented in a tool called MuTMuT within the general framework. Evaluation of MuTMuT on mutation testing of 12 multithreaded programs shows that it can substantially reduce the time required for mutation testing of multithreaded code.Copyright © 2012 John Wiley & Sons, Ltd.

[1]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[2]  Phil McMinn,et al.  Mutation Operators for Agent-Based Models , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[3]  John Chapin,et al.  Systematic Testing of Multithreaded Programs , 2000 .

[4]  Scott D. Stoller,et al.  Model-checking multi-threaded distributed Java programs , 2000, International Journal on Software Tools for Technology Transfer.

[5]  Francisco Vilar Brasileiro,et al.  Improving Automated Testing of Multi-threaded Software , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[6]  Viktor Kuncak,et al.  CrystalBall: Predicting and Preventing Inconsistencies in Deployed Distributed Systems , 2009, NSDI.

[7]  Sarfraz Khurshid,et al.  Test generation through programming in UDITA , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[8]  John A. Clark,et al.  The Rigorous Generation of Java Mutation Operators Using HAZOP , 1999 .

[9]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[10]  Alex Groce,et al.  Tackling Large Verification Problems with the Swarm Tool , 2008, SPIN.

[11]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[12]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009 .

[13]  S. Inglis,et al.  Jumble Java Byte Code to Measure the Effectiveness of Unit Tests , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[14]  Pravesh Kothari,et al.  A randomized scheduler with probabilistic guarantees of finding bugs , 2010, ASPLOS XV.

[15]  Eitan Farchi,et al.  Applications of synchronization coverage , 2005, PPoPP.

[16]  A. Jefferson Offutt,et al.  Constraint-Based Automatic Test Data Generation , 1991, IEEE Trans. Software Eng..

[17]  Harry T. Larson,et al.  On Whistle Blowing , 1971, Computer.

[18]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[19]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[20]  Grigore Rosu,et al.  Improved multithreaded unit testing , 2011, ESEC/FSE '11.

[21]  Yuanyuan Zhou,et al.  CTrigger: exposing atomicity violation bugs from their hiding places , 2009, ASPLOS.

[22]  Madan Musuvathi,et al.  Iterative context bounding for systematic testing of multithreaded programs , 2007, PLDI '07.

[23]  Nikolai Tillmann,et al.  Test generation via Dynamic Symbolic Execution for mutation testing , 2010, 2010 IEEE International Conference on Software Maintenance.

[24]  Mike Papadakis,et al.  Automatic Mutation Test Case Generation via Dynamic Symbolic Execution , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[25]  Klaus Havelund,et al.  Towards a framework and a benchmark for testing tools for multi‐threaded programs , 2007, Concurr. Comput. Pract. Exp..

[26]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[27]  Matthew B. Dwyer,et al.  Parallel Randomized State-Space Search , 2007, 29th International Conference on Software Engineering (ICSE'07).

[28]  Giuliano Antoniol,et al.  Automatic mutation test input data generation via ant colony , 2007, GECCO '07.

[29]  Richard H. Carver,et al.  Mutation-based testing of concurrent programs , 1993, Proceedings of IEEE International Test Conference - (ITC).

[30]  Sebastian Burckhardt,et al.  Line-up: a complete and automatic linearizability checker , 2010, PLDI '10.

[31]  Mahesh Viswanathan,et al.  Incremental state-space exploration for programs with dynamically allocated data , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[32]  Thomas Ball,et al.  Finding and Reproducing Heisenbugs in Concurrent Programs , 2008, OSDI.

[33]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[34]  Gregg Rothermel,et al.  Regression model checking , 2009, 2009 IEEE International Conference on Software Maintenance.

[35]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[36]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[37]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[38]  Darko Marinov,et al.  Mutation Operators for Actor Systems , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[39]  Yu Yang,et al.  Efficient Stateful Dynamic Partial Order Reduction , 2008, SPIN.

[40]  Koushik Sen,et al.  Race directed random testing of concurrent programs , 2008, PLDI '08.

[41]  Haoxiang Lin,et al.  MODIST: Transparent Model Checking of Unmodified Distributed Systems , 2009, NSDI.

[42]  William Pugh,et al.  Unit testing concurrent software , 2007, ASE.

[43]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[44]  Daniel Hoffman,et al.  Tool Support for Testing Concurrent Java Components , 2003, IEEE Trans. Software Eng..

[45]  Dawson R. Engler,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Cmc: a Pragmatic Approach to Model Checking Real Code , 2022 .

[46]  Darko Marinov,et al.  A Framework for State-Space Exploration of Java-Based Actor Programs , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[47]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[48]  Mike Papadakis,et al.  Towards automating the generation of mutation tests , 2010, AST '10.

[49]  Mark Harman,et al.  Higher Order Mutation Testing , 2009, Inf. Softw. Technol..

[50]  Orna Grumberg,et al.  Bounded Model Checking of Concurrent Programs , 2005, CAV.

[51]  Stephan Merz,et al.  Model Checking , 2000 .

[52]  Vladimir N. Fleyshgakker,et al.  Efficient mutation analysis: a new approach , 1994, ISSTA '94.

[53]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[54]  A. Jefferson Offutt,et al.  Inter-class mutation operators for Java , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[55]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[56]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[57]  Darko Marinov,et al.  MuTMuT: Efficient Exploration for Mutation Testing of Multithreaded Code , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[58]  John A. Clark,et al.  Class Mutation : Mutation Testing for Object-Oriented Programs , 2000 .

[59]  Patrice Godefroid,et al.  Model checking for programming languages using VeriSoft , 1997, POPL '97.

[60]  Koushik Sen,et al.  CUTE and jCUTE: Concolic Unit Testing and Explicit Path Model-Checking Tools , 2006, CAV.

[61]  Gul A. Agha,et al.  ACTORS - a model of concurrent computation in distributed systems , 1985, MIT Press series in artificial intelligence.

[62]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[63]  J. Dingel,et al.  Mutation Operators for Concurrent Java (J2SE 5.0) , 2006, Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006).

[64]  Darko Marinov,et al.  Automatic testing of software with structurally complex inputs , 2005 .

[65]  J. Dingel,et al.  Comparative Assessment of Testing and Model Checking Using Program Mutation , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[66]  A. Jefferson Offutt,et al.  Mutation analysis using mutant schemata , 1993, ISSTA '93.

[67]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.