Applying a Pairwise Coverage Criterion to Scenario-Based Testing

Some scenario-based testing approaches propose to express a test suite as a regular expression (called scenario or pattern). It specifies a set of valid sequences of operations in a abstract way. When the regular expression is "unfold", test sequences are obtained. Usually, the unfolding is done in an exhaustive way, which can result in a combinatorial explosion. In this article, we explore a pairwise coverage criterion to select a subset of test sequences satisfying the pattern, in order to decrease the number of test sequences. The originality of the approach lies in the fact that the pairwise criterion is applied to the instantiated method calls (and not on the parameters). We applied this strategy to generate unit tests for Java classes. The quality of the test suites is evaluated with a mutation analysis and compared to test suites randomly generated.

[1]  A. Jefferson Offutt,et al.  An evaluation of combination strategies for test case selection , 2006, Empirical Software Engineering.

[2]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[3]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[4]  Jeff Yu Lei,et al.  Practical Combinatorial Testing: Beyond Pairwise , 2008, IT Professional.

[5]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[6]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[7]  Laurie A. Williams,et al.  On guiding the augmentation of an automated test suite via mutation analysis , 2009, Empirical Software Engineering.

[8]  Imran Bashir,et al.  Testing Object-Oriented Software , 1999, Springer New York.

[9]  Francisco Fernández de Vega,et al.  Enabling Object Reuse on Genetic Programming-Based Approaches to Object-Oriented Evolutionary Testing , 2010, EuroGP.

[10]  Phyllis G. Frankl,et al.  Further empirical studies of test effectiveness , 1998, SIGSOFT '98/FSE-6.

[11]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[12]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[13]  Imran Bashir,et al.  Testing object-oriented software - life cycle solutions , 1999 .

[14]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[15]  Marcelo d'Amorim,et al.  An Empirical Comparison of Automated Generation and Classification Techniques for Object-Oriented Unit Testing , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[16]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[17]  Andrea Arcuri.  Longer is Better: On the Role of Test Sequence Length in Software Testing , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[18]  Jeff Yu Lei,et al.  An Empirical Comparison of Combinatorial and Random Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[19]  Myra B. Cohen,et al.  Interaction Coverage Meets Path Coverage by SMT Constraint Solving , 2009, TestCom/FATES.

[20]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[21]  Jeff Yu Lei,et al.  Combinatorial Methods for Event Sequence Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[22]  Lydie du Bousquet,et al.  Filtering TOBIAS Combinatorial Test Suites , 2004, FASE.

[23]  Clifford A. Shaffer.  Data Structures and Algorithm Analysis in Java , 2011 .

[24]  Baowen Xu,et al.  Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite , 2008, 2008 The Eighth International Conference on Quality Software.

[25]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[26]  Phyllis G. Frankl,et al.  All-uses vs mutation testing: An experimental comparison of effectiveness , 1997, J. Syst. Softw..

[27]  Shuang Wang,et al.  Comparison of Unit-Level Automated Test Generation Tools , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[28]  Robert L. Probert,et al.  A measure for component interaction test coverage , 2001, Proceedings ACS/IEEE International Conference on Computer Systems and Applications.

[29]  Nicholas Kalouptsidis,et al.  Efficient Algorithms for , 1999 .

[30]  Frédéric Dadeau,et al.  jSynoPSys - A Scenario-Based Testing Tool based on the Symbolic Animation of B Machines , 2009, Electron. Notes Theor. Comput. Sci..

[31]  Patrick J. Schroeder,et al.  Pairwise Testing : A Best Practice That Isn ’ , 2005 .

[32]  Jeff Yu Lei,et al.  Efficient Algorithms for T-way Test Sequence Generation , 2012, 2012 IEEE 17th International Conference on Engineering of Complex Computer Systems.

[33]  Yves Ledru,et al.  Directed random reduction of combinatorial test suites , 2007, RT '07.

[34]  Gordon Fraser,et al.  Testing Container Classes: Random or Systematic? , 2011, FASE.

[35]  Joel Greenyer,et al.  Synthesizing tests for combinatorial coverage of modal scenario specifications , 2015, 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS).

[36]  Elaine J. Weyuker,et al.  Comparison of program testing strategies , 1991, TAV4.

[37]  A. Jefferson Offutt,et al.  MuJava: a mutation system for java , 2006, ICSE.

[38]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[39]  Mark Allen Weiss,et al.  Data structures and algorithm analysis in Ada , 1993 .

[40]  Joachim Wegener,et al.  Evolutionary unit testing of object-oriented software using strongly-typed genetic programming , 2006, GECCO '06.

[41]  Rachel Tzoref,et al.  System Level Combinatorial Testing in Practice -- The Concurrent Maintenance Case Study , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[42]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[43]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[44]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[45]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[46]  Peter Zoeteweij,et al.  An Evaluation of Similarity Coefficients for Software Fault Localization , 2006, 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC'06).

[47]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[48]  Anna Gavling,et al.  The ART at , 2008 .

[49]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[50]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.