Testing across configurations: implications for combinatorial testing

User configurable software systems allow users to customize functionality at run time. In essence, each such system consists of a family of potentially thousands or millions of program instantiations. Testing methods cannot test all of these configurations, therefore some sampling mechanism must be applied. A common approach to providing such a mechanism has been to use combinatorial interaction testing. To date, however, little work has been done to quantify the effects of different configurations on a test suites' operation and effectiveness. In this paper we present a case study that investigates the effects of changing configurations on two types of test suites. Our results show that test coverage and fault detection effectiveness do not vary much across configurations for entire test suites; however, for individual test cases and certain types of faults, configurations matter.

[1]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[2]  M. Hardojo.  The effect of field data integrity on failure reproduction and fault isolation , 2004, ICSE 2004.

[3]  C. L. Mallows,et al.  Applying design of experiments to software testing: experience report , 1997, ICSE '97.

[4]  Ashish Jain,et al.  Model-based testing of a highly programmable system , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[5]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[6]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[7]  Alexander Pretschner,et al.  Model-Based Testing in Practice , 2005, FM.

[8]  R. L. Erickson,et al.  Improved quality of protocol testing through techniques of experimental design , 1994, Proceedings of ICC/SUPERCOMM'94 - 1994 International Conference on Communications.

[9]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[10]  D.M. Cohen,et al.  The Combinatorial Design Approach to Automatic Test Generation , 1996, IEEE Softw..

[11]  Elaine J. Weyuker.  Testing Component-Based Software: A Cautionary Tale , 1998, IEEE Softw..

[12]  M. J. Reilly,et al.  An investigation of the applicability of design of experiments to software testing , 2002, 27th Annual NASA Goddard/IEEE Software Engineering Workshop, 2002. Proceedings..

[13]  C. L. Mallows,et al.  Applying Design of Experiments to Software Testing , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[14]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[15]  Paul D. Seymour,et al.  AETG/sup SM/ Web: a Web based service for automatic efficient test generation from functional requirements , 1998, Proceedings. 2nd IEEE Workshop on Industrial Strength Formal Specification Techniques.

[16]  Gregg Rothermel,et al.  The impact of software evolution on code coverage information , 2001, Proceedings IEEE International Conference on Software Maintenance. ICSM 2001.

[17]  Barton P. Miller,et al.  An empirical study of the robustness of MacOS applications using random testing , 2006, RT '06.

[18]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[19]  Gregg Rothermel,et al.  Analyzing Regression Test Selection Techniques , 1996, IEEE Trans. Software Eng..

[20]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[21]  Douglas C. Schmidt,et al.  Skoll: distributed continuous quality assurance , 2004, Proceedings. 26th International Conference on Software Engineering.

[22]  K. Burr,et al.  Combinatorial Test Techniques : Table-based Automation , Test Generation and Code Coverage , 1998 .

[23]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.