Mutation Testing Approach to Negative Testing

Negative testing deals with an important problem of assessing a system ability to handle unexpected situations. Such situations, if unhandled, may lead to system failures that in some cases can have catastrophic consequences. This paper presents a mutation testing-based approach for generation of test cases supporting negative testing. Application of this approach can provide, in a systematic and human-unbiased way, test cases effectively testing wide range of unexpected situations. Thus, it can contribute to improvement of a tested system. The paper formally defines mutation operators used to control the generation process, describes a generic framework for the generation and execution of the test cases, and explains how to interpret results.

[1]  Gordon Fraser,et al.  Using Model-Checkers for Mutation-Based Test-Case Generation, Coverage Analysis and Specification Analysis , 2006, 2006 International Conference on Software Engineering Advances (ICSEA'06).

[2]  Joanna Strug,et al.  Machine Learning Approach in Mutation Testing , 2012, ICTSS.

[3]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[4]  Shing-Chi Cheung,et al.  Automated test case generation for the stress testing of multimedia systems , 2002, Softw. Pract. Exp..

[5]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[6]  Joanna Strug,et al.  Using Structural Similarity to Classify Tests in Mutation Testing , 2013 .

[7]  Laurie A. Williams,et al.  Should software testers use mutation analysis to augment a test set? , 2009, J. Syst. Softw..

[8]  J. Strug Classification of Mutation Operators Applied to Design Models , 2013 .

[9]  Lionel C. Briand,et al.  Stress testing real-time systems with genetic algorithms , 2005, GECCO '05.

[10]  Fevzi Belli,et al.  Finite state testing and analysis of graphical user interfaces , 2001, Proceedings 12th International Symposium on Software Reliability Engineering.

[11]  Fevzi Belli,et al.  Model-based mutation testing - Approach and case studies , 2016, Sci. Comput. Program..

[12]  Joanna Strug,et al.  Classifying Mutants with Decomposition Kernel , 2016, ICAISC.

[13]  Marin Litoiu,et al.  Model-based performance testing: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[14]  Anna Derezinska Object-oriented mutation to assess the quality of tests , 2003, 2003 Proceedings 29th Euromicro Conference.

[15]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[16]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[17]  Wolfgang Herzner,et al.  Fault-Based Generation of Test Cases from UML-Models - Approach and Some Experiences , 2011, SAFECOMP.

[18]  Inmaculada Medina-Bulo,et al.  Evolutionary mutation testing , 2011, Inf. Softw. Technol..

[19]  A. Jefferson Offutt,et al.  Inter-class mutation operators for Java , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[20]  Fevzi Belli,et al.  A Formal Framework for Mutation Testing , 2010, 2010 Fourth International Conference on Secure Software Integration and Reliability Improvement.

[21]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[22]  Joanna Strug Mutation Testing Approach to Evaluation of Design Models , 2013 .

[23]  James W. Fawcett,et al.  Measuring Component Specification-Implementation Concordance with Semantic Mutation Testing , 2011, CATA.

[24]  Bernhard K. Aichernig,et al.  Model-Based Mutation Testing of an Industrial Measurement Device , 2014, TAP@STAF.

[25]  Stuart Reid,et al.  An empirical analysis of equivalence partitioning, boundary value analysis and random testing , 1997, Proceedings Fourth International Software Metrics Symposium.

[26]  A. Jefferson Offutt,et al.  A logic mutation approach to selective mutation for programs and queries , 2011, Inf. Softw. Technol..

[27]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[28]  Vadim Okun,et al.  Mutation operators for specifications , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[29]  F. Belli,et al.  On “ Negative ” Tests of Web Applications , 2007 .

[30]  Bernhard K. Aichernig,et al.  Test case generation by OCL mutation and constraint solving , 2005, Fifth International Conference on Quality Software (QSIC'05).

[31]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[32]  Paulo César Masiero,et al.  Mutation testing applied to validate specifications based on statecharts , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[33]  Nikolai Tillmann,et al.  Test generation via Dynamic Symbolic Execution for mutation testing , 2010, 2010 IEEE International Conference on Software Maintenance.

[34]  Ying Jiang,et al.  An Approach to Testing Black-Box Components Using Contract-Based Mutation , 2008, Int. J. Softw. Eng. Knowl. Eng..

[35]  A.P. Mathur Performance, effectiveness, and reliability issues in software testing , 1991, [1991] Proceedings The Fifteenth Annual International Computer Software & Applications Conference.

[36]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[37]  Bernhard K. Aichernig,et al.  Killing strategies for model‐based mutation testing , 2015, Softw. Test. Verification Reliab..

[38]  Tao Xie,et al.  Automatic test generation for mutation testing on database applications , 2013, 2013 8th International Workshop on Automation of Software Test (AST).

[39]  Macario Polo,et al.  Mutation Testing Cost Reduction Techniques: A Survey , 2010, IEEE Software.