Coverage and adequacy in software product line testing

Software product line modeling has received a great deal of attention for its potential in fostering reuse of software artifacts across development phases. Research on the testing phase, has focused on identifying the potential for reuse of test cases across product line instances. While this offers potential reductions in test development effort for a given product line instance, it does not focus on and leverage the fundamental abstraction that is inherent in software product lines - variability.In this paper, we illustrate how rich software product line modeling notations can be mapped onto an underlying relational model that captures variability in the feasible product line instances. This relational model serves as the semantic basis for defining a family of coverage criteria for testing of a product line. These criteria make it possible to accumulate test coverage information for the product line itself over the course of multiple product line instance development efforts. Cumulative coverage, in turn, enables targeted testing efforts for new product line instances. We describe how combinatorial interaction testing methods can be applied to define test configurations that achieve a desired level of coverage and identify challenges to scaling such methods to large, complex software product lines.

[1]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[2]  C. L. Mallows,et al.  Applying Design of Experiments to Software Testing , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[3]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[4]  Don S. Batory,et al.  Feature Models, Grammars, and Propositional Formulas , 2005, SPLC.

[5]  Juha Taina,et al.  Hook and Template Coverage Criteria for Testing Framework-based Software Product Families , .

[6]  Klaus Pohl,et al.  Software Product Line Engineering , 2005 .

[7]  Klaus Pohl,et al.  Modelling requirements variability across product lines , 2005, 13th IEEE International Conference on Requirements Engineering (RE'05).

[8]  Henry Muccini,et al.  Towards Testing Product Line Architectures , 2003, TACoS.

[9]  Madhan Shridhar Phadke,et al.  Quality Engineering Using Robust Design , 1989 .

[10]  A. Hartman.  Software and Hardware Testing Using Combinatorial Covering Suites , 2005 .

[11]  Myra B. Cohen,et al.  A variable strength interaction testing of components , 2003, Proceedings 27th Annual International Computer Software and Applications Conference. COMPAC 2003.

[12]  C. L. Mallows,et al.  Applying design of experiments to software testing: experience report , 1997, ICSE '97.

[13]  Hassan Gomaa.  Designing Software Product Lines with UML 2.0: From Use Cases to Pattern-Based Software Architectures , 2006, ICSR.

[14]  Charles J. Colbourn,et al.  Prioritized interaction testing for pair-wise coverage with seeding and constraints , 2006, Inf. Softw. Technol..

[15]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[16]  Krzysztof Czarnecki,et al.  Formalizing cardinality-based feature models and their specialization , 2005, Softw. Process. Improv. Pract..

[17]  Robert L. Probert,et al.  A measure for component interaction test coverage , 2001, Proceedings ACS/IEEE International Conference on Computer Systems and Applications.

[18]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[19]  Alan Hartman,et al.  Problems and algorithms for covering arrays , 2004, Discret. Math..

[20]  Steven David Prestwich,et al.  Constraint-Based Approaches to the Covering Test Problem , 2004, CSCLP.

[21]  Henry Muccini,et al.  Using software architecture for code testing , 2004, IEEE Transactions on Software Engineering.

[22]  John D. McGregor,et al.  Testing a Software Product Line , 2001, PSSE.

[23]  Antonia Bertolino,et al.  Use case-based testing of product lines , 2003, ESEC/FSE-11.

[24]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[25]  Paul Clements,et al.  Software product lines - practices and patterns , 2001, SEI series in software engineering.

[26]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[27]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[28]  Henry Muccini,et al.  Deriving test plans from architectural descriptions , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[29]  Myra B. Cohen,et al.  Constructing test suites for interaction testing , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[30]  John Hatcliff,et al.  CALM and Cadena: metamodeling for component-based product-line development , 2006, Computer.

[31]  Alexander L. Wolf,et al.  Software testing at the architectural level , 1996, ISAW '96.

[32]  David Garlan,et al.  A formal basis for architectural connection , 1997, TSEM.