Model-based, mutation-driven test case generation via heuristic-guided branching search

This work introduces a heuristic-guided branching search algorithm for model-based, mutation-driven test case generation. The algorithm is designed towards the efficient and computationally tractable exploration of discrete, non-deterministic models with huge state spaces. Asynchronous parallel processing is a key feature of the algorithm. The algorithm is inspired by the successful path planning algorithm Rapidly exploring Random Trees (RRT). We adapt RRT in several aspects towards test case generation. Most notably, we introduce parametrized heuristics for start and successor state selection, as well as a mechanism to construct test cases from the data produced during search. We implemented our algorithm in the existing test case generation framework MoMuT. We present an extensive evaluation of our heuristics and parameters based on a diverse set of demanding models obtained in an industrial context. In total we continuously utilized 128 CPU cores on three servers for two weeks to gather the experimental data presented. Using statistical methods we determine which heuristics are performing well on all models. With our new algorithm, we are now able to process models consisting of over 2300 concurrent objects. To our knowledge there is no other mutation driven test case generation tool that is able to process models of this magnitude.

[1]  Gordon Fraser,et al.  Evolutionary Generation of Whole Test Suites , 2011, 2011 11th International Conference on Quality Software.

[2]  Bernhard K. Aichernig,et al.  MoMut::UML Model-Based Mutation Testing for UML , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[3]  Gordon Fraser,et al.  Combining search-based and constraint-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[4]  Yvan Labiche,et al.  A systematic review of state-based test tools , 2013, International Journal on Software Tools for Technology Transfer.

[5]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[6]  Hiroyuki Sato,et al.  GRT: Program-Analysis-Guided Random Testing (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[7]  Bernhard K. Aichernig,et al.  Killing strategies for model‐based mutation testing , 2015, Softw. Test. Verification Reliab..

[8]  Kaisa Sere,et al.  An Approach to Object-Orientation in Action Systems , 1998, MPC.

[9]  John A. Clark,et al.  Search-based mutation testing for Simulink models , 2005, GECCO '05.

[10]  Ralph-Johan Back,et al.  Decentralization of Process Nets with Centralized Control , 1983, PODC.

[11]  R. Lipton,et al.  Mutation analysis , 1998 .

[12]  Jan Peleska,et al.  Model-Based Testing Against Complex SysML Models , 2015, SyDe Summer School.

[13]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[14]  Bernhard K. Aichernig,et al.  Mapping UML to Labeled Transition Systems for Test-Case Generation - A Translation via Object-Oriented Action Systems , 2009, FMCO.

[15]  Vikram S. Adve,et al.  LLVM: a compilation framework for lifelong program analysis & transformation , 2004, International Symposium on Code Generation and Optimization, 2004. CGO 2004..

[16]  E. LESTER SMITH,et al.  AND OTHERS , 2005 .

[17]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[18]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[19]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[20]  Steven M. LaValle,et al.  RRT-connect: An efficient approach to single-query path planning , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[21]  C. Martin.  2015 , 2015, Les 25 ans de l’OMC: Une rétrospective en photos.

[22]  Yvan Labiche,et al.  A Systematic Review of Model Based Testing Tool Support , 2010 .

[23]  Rajesh Subramanyan,et al.  A survey on model-based testing approaches: a systematic review , 2007, WEASELTech '07.

[24]  Alessandra Gorla,et al.  Search-based data-flow test generation , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[25]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[26]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[27]  V. Lakshmi Narasimhan,et al.  ADTEST: A Test Data Generation Suite for Ada Software Systems , 1997, IEEE Trans. Software Eng..

[28]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[29]  Bertrand Meyer,et al.  ARTOO: adaptive random testing for object-oriented software , 2008, ICSE.

[30]  Joachim Wegener,et al.  Automatic Test Data Generation For Structural Testing Of Embedded Software Systems By Evolutionary Testing , 2002, GECCO.

[31]  Bernhard K. Aichernig,et al.  Model-Based Mutation Testing of an Industrial Measurement Device , 2014, TAP@STAF.

[32]  John Aycock,et al.  A brief history of just-in-time , 2003, CSUR.

[33]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[34]  Alexander Pretschner,et al.  One evaluation of model-based testing and its automation , 2005, ICSE.

[35]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[36]  Bernhard K. Aichernig,et al.  Model-based mutation testing via symbolic refinement checking , 2015, Sci. Comput. Program..

[37]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[38]  Thierry Jéron,et al.  TGV : theory , principles and algorithms A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2004 .

[39]  Manfred Broy,et al.  Model-Based Testing of Reactive Systems, Advanced Lectures , 2005 .

[40]  Edsger W. Dijkstra,et al.  Guarded commands, nondeterminacy and formal derivation of programs , 1975, Commun. ACM.

[41]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[42]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[43]  S. LaValle.  Rapidly-exploring random trees : a new tool for path planning , 1998 .

[44]  James Kapinski,et al.  Efficient Guiding Strategies for Testing of Temporal Properties of Hybrid Systems , 2015, NFM.

[45]  S. Ramesh,et al.  Randomized directed testing (REDIRECT) for Simulink/Stateflow models , 2008, EMSOFT '08.

[46]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[47]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[48]  John A. Clark,et al.  An automated framework for structural test-data generation , 1998, Proceedings 13th IEEE International Conference on Automated Software Engineering (Cat. No.98EX239).