Boundary value analysis in automatic white-box test generation

White-box testing is an effective technique for generating test cases to provide high coverage for programs. We usually select several execution paths using some strategy, and generate a corresponding test case that follows each path. Each execution path corresponds to an input subspace bounded by constraints. Extreme values in these subspaces are very likely to trigger failures since the constraints bounding input subspaces may be faulty. In this paper, we propose a new way of defining the boundaries of comparison predicates in white-box testing, and apply constrained combinatorial testing to cover these boundaries with reduced number of test cases. Our approach can guarantee to cover all possible boundaries for each selected execution path, thus achieving high fault coverage for boundary faults, while the original structural coverage is still preserved.

[1]  Bruno Dutertre,et al.  A Fast Linear-Arithmetic Solver for DPLL(T) , 2006, CAV.

[2]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[3]  Gordon Fraser,et al.  Generating Test Suites with Augmented Dynamic Symbolic Execution , 2013, TAP@STAF.

[4]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[5]  Paul Ammann,et al.  Using formal methods to derive test frames in category-partition testing , 1994, Proceedings of COMPASS'94 - 1994 IEEE 9th Annual Conference on Computer Assurance.

[6]  Jian Zhang,et al.  Path-oriented test data generation using symbolic execution and constraint solving techniques , 2004, Proceedings of the Second International Conference on Software Engineering and Formal Methods, 2004. SEFM 2004..

[7]  John A. Clark,et al.  Automatic Test Data Generation for Multiple Condition and MCDC Coverage , 2009, 2009 Fourth International Conference on Software Engineering Advances.

[8]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[9]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[10]  Myra B. Cohen,et al.  An Improved Meta-heuristic Search for Constrained Interaction Testing , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[11]  Yong Zhao,et al.  Cascade: A Test Generation Tool for Combinatorial Testing , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[12]  Baowen Xu,et al.  Comparing logic coverage criteria on test case prioritization , 2012, Science China Information Sciences.

[13]  Jun Yan,et al.  Automatic Test Data Generation for Unit Testing to Achieve MC/DC Criterion , 2014, 2014 Eighth International Conference on Software Security and Reliability.

[14]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[15]  Jian Zhang,et al.  Automatic Generation of Combinatorial Test Data , 2014, SpringerBriefs in Computer Science.

[16]  Chin-Yu Huang,et al.  A Study of Enhanced MC/DC Coverage Criterion for Software Testing , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[17]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[18]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[19]  Bruno Legeard,et al.  Boundary coverage criteria for test generation from formal models , 2004, 15th International Symposium on Software Reliability Engineering.

[20]  Jeff Yu Lei,et al.  IPOG/IPOG‐D: efficient test generation for multi‐way combinatorial testing , 2008, Softw. Test. Verification Reliab..

[21]  G. S. Prashanth,et al.  Increase in Modified Condition/Decision Coverage using program code transformer , 2013, 2013 3rd IEEE International Advance Computing Conference (IACC).

[22]  Jacek Czerwonka,et al.  Pairwise Testing in Real World , 2006 .

[23]  David L. Dill,et al.  A Decision Procedure for Bit-Vectors and Arrays , 2007, CAV.

[24]  Yu Lei,et al.  Introduction to Combinatorial Testing , 2013 .

[25]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[26]  Jian Zhang,et al.  A Test Data Generation Tool for Unit Testing of C Programs , 2006, 2006 Sixth International Conference on Quality Software (QSIC'06).

[27]  Jeff Yu Lei,et al.  An Efficient Algorithm for Constraint Handling in Combinatorial Test Generation , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[28]  Jun Yan,et al.  Generating combinatorial test suite using combinatorial optimization , 2014, J. Syst. Softw..

[29]  Nikolai Tillmann,et al.  Guided test generation for coverage criteria , 2010, 2010 IEEE International Conference on Software Maintenance.

[30]  Giuliano Antoniol,et al.  MC/DC automatic test input data generation , 2009, GECCO '09.

[31]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[32]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[33]  Kamran Ghani,et al.  Searching for Test Data , 2009 .

[34]  J Hayhurst Kelly,et al.  A Practical Tutorial on Modified Condition/Decision Coverage , 2001 .

[35]  Armin Biere,et al.  Boolector: An Efficient SMT Solver for Bit-Vectors and Arrays , 2009, TACAS.

[36]  Tim King,et al.  CVC 4 , 2011 .

[37]  Bruno Marre,et al.  PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis , 2005, EDCC.

[38]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.