Improving Failure Detection by Automatically Generating Test Cases Near the Boundaries

Boundary value analysis is a typical conventional testing technique. However, manually identifying input regions and writing test cases are labor-intensive and time-consuming. In this paper, we propose a search-based random testing approach, which automatically generates test data along the boundaries of semantic regions of the input domain. The experiments on mutated programs confirm the effectiveness and efficiency of the proposed approach. Furthermore, our approach significantly outperforms the conventional ART (Adaptive Random Testing) methods, which sample test cases evenly across the input regions. Our approach also outperforms EvoSuite, a state-of-the-art tool that generates test cases satisfying certain coverage criterion.

[1]  Stuart Reid,et al.  An empirical analysis of equivalence partitioning, boundary value analysis and random testing , 1997, Proceedings Fourth International Software Metrics Symposium.

[2]  Gordon Fraser,et al.  Generating Test Suites with Augmented Dynamic Symbolic Execution , 2013, TAP@STAF.

[3]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[4]  Nikolai Tillmann,et al.  Guided test generation for coverage criteria , 2010, 2010 IEEE International Conference on Software Maintenance.

[5]  Gary McGraw,et al.  Generating Software Test Data by Evolution , 2001, IEEE Trans. Software Eng..

[6]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[7]  Pablo Moscato,et al.  3 Memetic Algorithms , 2004 .

[8]  Nashat Mansour,et al.  Simulated Annealing and Genetic Algorithms for Optimal Regression Testing , 1999, J. Softw. Maintenance Res. Pract..

[9]  MaYu-Seung,et al.  MuJava: an automated class mutation system , 2005 .

[10]  Kay Chen Tan,et al.  A Multi-Facet Survey on Memetic Computation , 2011, IEEE Transactions on Evolutionary Computation.

[11]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[12]  Mark Harman,et al.  A multi-objective approach to search-based test data generation , 2007, GECCO '07.

[13]  Bertrand Meyer,et al.  ARTOO , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[14]  Tsong Yueh Chen,et al.  Adaptive Random Testing: The ART of test case diversity , 2010, J. Syst. Softw..

[15]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[16]  Pablo Moscato,et al.  On Evolution, Search, Optimization, Genetic Algorithms and Martial Arts : Towards Memetic Algorithms , 1989 .

[17]  Roy P. Pargas,et al.  Test‐data generation using genetic algorithms , 1999 .

[18]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[19]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[20]  I. K. Mak,et al.  Adaptive Random Testing , 2004, ASIAN.

[21]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[22]  Gordon Fraser,et al.  EvoSuite at the SBST 2013 Tool Competition , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[23]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..

[24]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[25]  James Miller,et al.  Centroidal Voronoi Tessellations- A New Approach to Random Testing , 2013, IEEE Transactions on Software Engineering.

[26]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[27]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[28]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[29]  Tsong Yueh Chen,et al.  Mirror adaptive random testing , 2004, Inf. Softw. Technol..

[30]  Emmanuel Letier,et al.  Simulating and optimising design decisions in quantitative goal models , 2011, 2011 IEEE 19th International Requirements Engineering Conference.

[31]  Lionel C. Briand,et al.  Adaptive random testing: an illusion of effectiveness? , 2011, ISSTA '11.

[32]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[33]  L. Lundberg,et al.  Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making , 2005 .

[34]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[35]  Takahide Yoshikawa,et al.  Random program generator for Java JIT compiler test system , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[36]  Bruno Legeard,et al.  Automated Boundary Testing from Z and B , 2002, FME.

[37]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.