The Minimal Failure-Causing Schema of Combinatorial Testing

Combinatorial Testing (CT) involves the design of a small test suite to cover the parameter value combinations so as to detect failures triggered by the interactions among these parameters. To make full use of CT and to extend its advantages, this article first gives a model of CT and then presents a theory of the Minimal Failure-causing Schema (MFS), including the concept of the MFS, proof of its existence, some of its properties, and a method of finding the MFS. Then we propose a methodology for CT based on this MFS theory and the existing research. Our MFS-based methodology emphasizes that CT should work on accurate testing requirements, and has the following advantages: 1) Detect failure to the greatest degree with the least cost. 2) Effectiveness is improved by emphasizing mining of the information in software and making full use of the information gained from test design and execution. 3) Determine the root causes of failures and reveal related faults near the exposed ones. 4) Provide a foundation and model for regression testing and software quality evaluation of CT. A case study is presented to illustrate the MFS-based CT methodology, and an empirical study on a real software developed by us shows that the MFS really exists and the methodology based on MFS can considerably improve CT.

[1]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[2]  Jeff Yu Lei,et al.  IPOG: A General Strategy for T-Way Software Testing , 2007, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07).

[3]  Baowen Xu,et al.  Test Plan Design for Software Configuration Testing , 2003, Software Engineering Research and Practice.

[4]  Moataz A. Ahmed,et al.  Pair-wise test coverage using genetic algorithms , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[5]  Tatsuhiro Tsuchiya,et al.  Using artificial life techniques to generate test cases for combinatorial testing , 2004, Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004..

[6]  Robert L. Probert,et al.  A measure for component interaction test coverage , 2001, Proceedings ACS/IEEE International Conference on Computer Systems and Applications.

[7]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[8]  Robert L. Probert,et al.  Formulation of the Interaction Test Coverage Problem as an Integer Program , 2002, TestCom.

[9]  Jonas Mellin,et al.  Handling Constraints in the Input Space when Using Combination Strategies for Software Testing , 2006 .

[10]  Jeff Offutt,et al.  Input parameter modeling for combination strategies , 2007, ICSE 2007.

[11]  Robert L. Probert,et al.  A practical strategy for testing pair-wise coverage of network interfaces , 1996, Proceedings of ISSRE '96: 7th International Symposium on Software Reliability Engineering.

[12]  Baowen Xu,et al.  A New Heuristic for Test Suite Generation for Pair-wise Testing , 2006, SEKE.

[13]  J. Czerwonka.  Pairwise Testing in the Real World : Practical Extensions to Test-Case Scenarios , 2011 .

[14]  James A. Whittaker,et al.  A software testing model: using design of experiments (doe) and logistic regression , 2001 .

[15]  Baowen Xu,et al.  Automatic Test Generation for N-Way Combinatorial Testing , 2005, QoSA/SOQUA.

[16]  Kari J. Nurmela,et al.  Upper bounds for covering arrays by tabu search , 2004, Discret. Appl. Math..

[17]  Myra B. Cohen,et al.  Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach , 2008, IEEE Transactions on Software Engineering.

[18]  Alan W. Williams,et al.  Determination of Test Configurations for Pair-Wise Interaction Coverage , 2000, TestCom.

[19]  R. Krishnan,et al.  Combinatorial testing: learnings from our experience , 2007, SOEN.

[20]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[21]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[22]  Baowen Xu,et al.  Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite , 2008, 2008 The Eighth International Conference on Quality Software.

[23]  Jiangfan Shi,et al.  Exploiting Constraint Solving History to Construct Interaction Test Suites , 2007, Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007).

[24]  Patrick J. Schroeder,et al.  Pairwise Testing : A Best Practice That Isn ’ , 2005 .

[25]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[26]  Myra B. Cohen,et al.  Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization , 2007, 2007 IEEE International Conference on Software Maintenance.

[27]  Charles J. Colbourn,et al.  Test prioritization for pairwise interaction coverage , 2005, A-MOST.

[28]  Alexander Pretschner,et al.  Model-Based Testing in Practice , 2005, FM.

[29]  Shi Liang,et al.  Generating Optimal Test Set for Neighbor Factors Combinatorial Testing , 2006, 2006 Sixth International Conference on Quality Software (QSIC'06).

[30]  Charles J. Colbourn,et al.  Algorithms for covering arrays , 2006 .

[31]  Andrew B. Kahng,et al.  Combinatorial group testing methods for the BIST diagnosis problem , 2004, ASP-DAC 2004: Asia and South Pacific Design Automation Conference 2004 (IEEE Cat. No.04EX753).

[32]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[33]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[34]  Colin L. Mallows,et al.  Factor-covering designs for testing software , 1998 .

[35]  Yu-Wen Tung,et al.  Automating test case generation for the new generation mission software system , 2000, 2000 IEEE Aerospace Conference. Proceedings (Cat. No.00TH8484).

[36]  Myra B. Cohen,et al.  Designing Test Suites for Software Interactions Testing , 2004 .

[37]  Myra B. Cohen,et al.  Augmenting simulated annealing to build interaction test suites , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[38]  Soumen Maity,et al.  Improved test generation algorithms for pair-wise testing , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[39]  Steven David Prestwich,et al.  Constraint Models for the Covering Test Problem , 2006, Constraints.

[40]  Charles J. Colbourn,et al.  Locating and detecting arrays for interaction faults , 2008, J. Comb. Optim..

[41]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[42]  Bogdan Korel,et al.  Generating expected results for automated black-box testing , 2002, Proceedings 17th IEEE International Conference on Automated Software Engineering,.

[43]  Ashish Jain,et al.  Modeling requirements for combinatorial software testing , 2005, A-MOST.

[44]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[45]  Steven David Prestwich,et al.  Constraint-Based Approaches to the Covering Test Problem , 2004, CSCLP.

[46]  Andrew B. Kahng,et al.  Combinatorial group testing methods for the BIST diagnosis problem , 2004 .

[47]  Baowen Xu,et al.  A Software Debugging Method Based on Pairwise Testing , 2005, International Conference on Computational Science.

[48]  Angelo Gargantini,et al.  A Logic-Based Approach to Combinatorial Testing with Constraints , 2008, TAP.

[49]  Myra B. Cohen,et al.  Towards incremental adaptive covering arrays , 2007, ESEC-FSE '07.

[50]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[51]  Noritaka Kobayashi,et al.  Design and Evaluation of Automatic Test Generation Strategies for Functional Testing of Software , 2002 .

[52]  Jerry Huller.  Reducing Time to Market with Combinatorial Design Method Testing , 2002 .

[53]  Patrick J. Schroeder,et al.  Comparing the fault detection effectiveness of n-way and random test suites , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[54]  Bogdan Korel,et al.  Black-box test reduction using input-output analysis , 2000, ISSTA '00.

[55]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[56]  Charles J. Colbourn,et al.  Covering arrays of higher strength from permutation vectors , 2006 .

[57]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[58]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[59]  Myra B. Cohen,et al.  Testing across configurations: implications for combinatorial testing , 2006, SOEN.

[60]  Baowen Xu,et al.  Generating combinatorial test suite for interaction relationship , 2007, SOQUA '07.

[61]  Renée C. Bryce,et al.  A framework of greedy methods for constructing interaction test suites , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[62]  James A. Whittaker,et al.  Prediction of software failures through logistic regression , 2004, Inf. Softw. Technol..

[63]  Tatsuhiro Tsuchiya,et al.  A new method for constructing pair-wise covering designs for software testing , 2002, Information Processing Letters.

[64]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[65]  Baowen Xu,et al.  A Browser Compatibility Testing Method Based on Combinatorial Testing , 2003, ICWE.

[66]  Ashish Jain,et al.  Model-based testing of a highly programmable system , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[67]  Glenford J. Myers,et al.  The art of software testing (2. ed.) , 2004 .

[68]  A. Hartman.  Software and Hardware Testing Using Combinatorial Covering Suites , 2005 .

[69]  Per Runeson,et al.  Efficient Evaluation of Multifactor Dependent System Performance Using Fractional Factorial Design , 2003, IEEE Trans. Software Eng..

[70]  M. Fredman,et al.  New techniques for designing qualitatively independent systems , 1998 .

[71]  Christine T. Cheng,et al.  Generating small combinatorial test suites to cover input-output relationships , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[72]  Myra B. Cohen,et al.  Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces , 2006, IEEE Trans. Software Eng..

[73]  Myra B. Cohen,et al.  Towards incremental adaptive covering arrays , 2007, ESEC-FSE companion '07.

[74]  Angelo Gargantini,et al.  Using SRI SAL model checker for combinatorial tests generation in the presence of temporal constraints , 2008, FM'08 2008.

[75]  Charles J. Colbourn,et al.  Test prioritization for pairwise interaction coverage , 2005, ACM SIGSOFT Softw. Eng. Notes.

[76]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[77]  A. Jefferson Offutt,et al.  An evaluation of combination strategies for test case selection , 2006, Empirical Software Engineering.

[78]  Xu Baowen,et al.  Generating combinatorial test suite for interaction relationship , 2007 .

[79]  Siddhartha R. Dalal,et al.  The Automatic Efficient Test Generator (AETG) system , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[80]  Alan W. Williams,et al.  Software component interaction testing: coverage measurement and generation of configurations , 2002 .

[81]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[82]  Gadiel Seroussi,et al.  Vector sets for exhaustive testing of logic circuits , 1988, IEEE Trans. Inf. Theory.

[83]  Siddhartha R. Dalal,et al.  Model-based testing in practice , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[84]  Adrian Dumitrescu,et al.  Efficient Algorithms for Generation of Combinatorial Covering Suites , 2003, ISAAC.

[85]  Paul Denny,et al.  Case studies and new results in combinatorial enumeration , 2000 .

[86]  Lee J. White,et al.  Generating test cases for GUI responsibilities using complete interaction sequences , 2000, Proceedings 11th International Symposium on Software Reliability Engineering. ISSRE 2000.

[87]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[88]  Andreas Blass,et al.  Pairwise Testing , 2002, Bull. EATCS.