The density algorithm for pairwise interaction testing

There are many published algorithms for generating interaction test suites for software testing, exemplified by AETG, IPO, TCG, TConfig, simulated annealing and other heuristic search, and combinatorial design techniques. Among these, greedy one‐test‐at‐a‐time methods (such as AETG and TCG) have proven to be a reasonable compromise between the needs for small test suites, fast test‐suite generation, and flexibility to accommodate a variety of testing scenarios. However, such methods suffer from the lack of a worst‐case logarithmic guarantee on test suite size, while methods that provide such a guarantee at present are less efficient or flexible, or do not produce test suites that are competitive in size for practical testing scenarios. In this paper, a new algorithm establishes that efficient, greedy, one‐test‐at‐a‐time methods can indeed produce a logarithmic worst‐case guarantee on the test suite size. In addition, this can be done while still producing test suites that are of competitive size, and in a time that is comparable to the published methods. It is deterministic, guaranteeing reproducibility. It generates only one candidate test at a time, permits users to ‘seed’ the test suite with specified tests, and allows users to specify constraints of combinations that should be avoided. Further, statistical analysis examines the impact of five variables used to tune this density algorithm for execution time and test suite size: weighting of density for factors, scaling of density, tie‐breaking, use of multiple candidates, and multiple repetitions using randomization. Copyright © 2007 John Wiley & Sons, Ltd.

[1]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[2]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[3]  Lj Lazić,et al.  Applying simulation and design of experiments to the embedded software testing process: Research Articles , 2004 .

[4]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[5]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[6]  Myra B. Cohen,et al.  Constructing strength three covering arrays with augmented annealing , 2003, Discret. Math..

[7]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[8]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[9]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[10]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[11]  Priya Kohli ORTHOGONAL ARRAYS , 2008 .

[12]  Per Runeson,et al.  Efficient Evaluation of Multifactor Dependent System Performance Using Fractional Factorial Design , 2003, IEEE Trans. Software Eng..

[13]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[14]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[15]  Christine T. Cheng,et al.  Generating small combinatorial test suites to cover input-output relationships , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[16]  B. Stevens,et al.  Covering arrays with mixed alphabet sizes , 2003 .

[17]  Robert L. Probert,et al.  A measure for component interaction test coverage , 2001, Proceedings ACS/IEEE International Conference on Computer Systems and Applications.

[18]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[19]  Adrian Dumitrescu,et al.  Efficient Algorithms for Generation of Combinatorial Covering Suites , 2003, ISAAC.

[20]  Lee J. White,et al.  Generating test cases for GUI responsibilities using complete interaction sequences , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[21]  C. L. Mallows,et al.  Applying Design of Experiments to Software Testing , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[22]  C. L. Mallows,et al.  Applying design of experiments to software testing: experience report , 1997, ICSE '97.

[23]  Robert F. Berry Computer Benchmark Evaluation and Design of Experiments, a Case Study , 1992, IEEE Trans. Computers.

[24]  N. J. A. Sloane,et al.  Tables of Orthogonal Arrays , 1999 .

[25]  Myra B. Cohen,et al.  A deterministic density algorithm for pairwise interaction coverage , 2004, IASTED Conf. on Software Engineering.

[26]  Alexander Pretschner,et al.  Model-Based Testing in Practice , 2005, FM.

[27]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[28]  Dusan M. Velasevic,et al.  Applying simulation and design of experiments to the embedded software testing process , 2004, Softw. Test. Verification Reliab..

[29]  Donald L. Kreher,et al.  On the state of strength‐three covering arrays , 2002 .

[30]  C. Colbourn Combinatorial aspects of covering arrays , 2006 .

[31]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[32]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[33]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[34]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[35]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[36]  Mary Lou Soffa,et al.  Regression testing of GUIs , 2003, ESEC/FSE-11.

[37]  Tatsuhiro Tsuchiya,et al.  A new method for constructing pair-wise covering designs for software testing , 2002, Information Processing Letters.

[38]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[39]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[40]  N. Sloane Covering arrays and intersecting codes , 1993 .

[41]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[42]  Willem H. Haemers,et al.  Equitable resolvable coverings , 2001 .

[43]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[44]  Myra B. Cohen,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[45]  A. Hartman Software and Hardware Testing Using Combinatorial Covering Suites , 2005 .