Coverage metrics for requirements-based testing

In black-box testing, one is interested in creating a suite of tests from requirements that adequately exercise the behavior of a software system without regard to the internal structure of the implementation. In current practice, the adequacy of black box test suites is inferred by examining coverage on an executable artifact, either source code or a software model.In this paper, we define structural coverage metrics directly on high-level formal software requirements. These metrics provide objective, implementation-independent measures of how well a black-box test suite exercises a set of requirements. We focus on structural coverage criteria on requirements formalized as LTL properties and discuss how they can be adapted to measure finite test cases. These criteria can also be used to automatically generate a requirements-based test suite. Unlike model or code-derived test cases, these tests are immediately traceable to high-level requirements. To assess the practicality of our approach, we apply it on a realistic example from the avionics domain.

[1]  Orna Kupferman,et al.  Resets vs. Aborts in Linear Temporal Logic , 2003, TACAS.

[2]  Angelo Gargantini,et al.  Using model checking to generate tests from requirements specifications , 1999, ESEC/FSE-7.

[3]  William E. Perry.  A standard for testing application software , 1986 .

[4]  David Harel,et al.  Come, let's play - scenario-based programming using LSCs and the play-engine , 2003 .

[5]  Loe M. G. Feijs,et al.  Test Generation for Intelligent Networks Using Model Checking , 1997, TACAS.

[6]  Sanjai Rayadurgam,et al.  Generating MC/DC adequate test sequences through model checking , 2003, 28th Annual NASA Goddard Software Engineering Workshop, 2003. Proceedings..

[7]  Robert P. Kurshan,et al.  A Practical Approach to Coverage in Model Checking , 2001, CAV.

[8]  Orna Kupferman,et al.  Coverage metrics for temporal logic model checking* , 2006, Formal Methods Syst. Des..

[9]  Steve Sims,et al.  TAME: A PVS Interface to Simplify Proofs for Automata Models , 1998 .

[10]  Mats Per Erik Heimdahl,et al.  Model checking RSML/sup -e/ requirements , 2002, 7th IEEE International Symposium on High Assurance Systems Engineering, 2002. Proceedings..

[11]  Nancy G. Leveson,et al.  Completeness and Consistency in Hierarchical State-Based Requirements , 1996, IEEE Trans. Software Eng..

[12]  Constance L. Heitmeyer,et al.  Model Checking Complete Requirements Specifications Using Abstraction , 2004, Automated Software Engineering.

[13]  Ajitha Rajan,et al.  Model Validation using Automatically Generated Requirements-Based Tests , 2007 .

[14]  Edmund M. Clarke,et al.  Model checking and abstraction , 1994, TOPL.

[15]  Hélène Waeselynck,et al.  Property-oriented testing: a strategy for exploring dangerous scenarios , 2003, SAC '03.

[16]  Paul Ammann,et al.  Using model checking to generate tests from specifications , 1998, Proceedings Second International Conference on Formal Engineering Methods (Cat.No.98EX241).

[17]  Fabio Somenzi,et al.  Vacuum Cleaning CTL Formulae , 2002, CAV.

[18]  J Hayhurst Kelly,et al.  A Practical Tutorial on Modified Condition/Decision Coverage , 2001 .

[19]  Ilan Beer,et al.  Efficient Detection of Vacuity in Temporal Model Checking , 2001, Formal Methods Syst. Des..

[20]  Mats Per Erik Heimdahl,et al.  Proving the shalls , 2003, International Journal on Software Tools for Technology Transfer.

[21]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[22]  Stephan Merz,et al.  Model Checking , 2000 .

[23]  Orna Kupferman,et al.  Aborts vs Resets in Linear Temporal Logic , 2002 .

[24]  Pascal Raymond,et al.  The synchronous data flow programming language LUSTRE , 1991, Proc. IEEE.

[25]  Orna Kupferman,et al.  Coverage metrics for formal verification , 2003, International Journal on Software Tools for Technology Transfer.

[26]  Massimiliano Di Penta,et al.  Assessing and improving state-based class testing: a series of experiments , 2004, IEEE Transactions on Software Engineering.

[27]  P. Caspi,et al.  A methodology for proving control systems with Lustre and PVS , 1999, Dependable Computing for Critical Applications 7.

[28]  Constance L. Heitmeyer,et al.  SCR: a toolset for specifying and analyzing requirements , 1995, COMPASS '95 Proceedings of the Tenth Annual Conference on Computer Assurance Systems Integrity, Software Safety and Process Security'.

[29]  Mats Per Erik Heimdahl,et al.  Specification test coverage adequacy criteria = specification test generation inadequacy criteria , 2004, Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings..

[30]  Michael W. Whalen,et al.  A formal semantics for RSML- e , 2000 .

[31]  Fabio Somenzi,et al.  Dos and don'ts of CTL state coverage estimation , 2003, DAC '03.

[32]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[33]  R Fisher,et al.  Design of Experiments , 1936 .

[34]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[35]  Sanjai Rayadurgam,et al.  Coverage based test-case generation using model checkers , 2001, Proceedings. Eighth Annual IEEE International Conference and Workshop On the Engineering of Computer-Based Systems-ECBS 2001.

[36]  Orna Kupferman,et al.  Vacuity Detection in Temporal Model Checking , 1999, CHARME.

[37]  Insup Lee,et al.  Specification-based testing with linear temporal logic , 2004, Proceedings of the 2004 IEEE International Conference on Information Reuse and Integration, 2004. IRI 2004..

[38]  Sigrid Eldh.  Software Testing Techniques , 2007 .

[39]  Timothy Kam,et al.  Coverage estimation for symbolic model checking , 1999, DAC '99.

[40]  Sanjai Rayadurgam.  Automated test-data generation from formal models of software , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[41]  Abdesselam Lakehal,et al.  Structural test coverage criteria for lustre programs , 2005, FMICS '05.

[42]  Sanjai Rayadurgam,et al.  Test-sequence generation from formal requirement models , 2001, Proceedings Sixth IEEE International Symposium on High Assurance Systems Engineering. Special Topic: Impact of Networking.

[43]  Willem Visser,et al.  Specification Centered Testing , 2004 .

[44]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[45]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[46]  Mats Per Erik Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[47]  Shaoying Liu,et al.  Criteria for generating specification-based tests , 1999, Proceedings Fifth IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'99) (Cat. No.PR00434).

[48]  Insup Lee,et al.  Data flow testing as model checking , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[49]  William C. Hetzel,et al.  The complete guide to software testing , 1984 .

[50]  Brani Vidakovic,et al.  Nonparametric Statistics with Applications to Science and Engineering (Wiley Series in Probability and Statistics) , 2007 .

[51]  Orna Grumberg,et al.  Model checking and modular verification , 1994, TOPL.

[52]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[53]  C. Eisner,et al.  Efficient Detection of Vacuity in ACTL Formulaas , 1997, CAV.

[54]  Insup Lee,et al.  A Temporal Logic Based Theory of Test Coverage and Generation , 2002, TACAS.

[55]  Dana Fisman,et al.  Reasoning with Temporal Logic on Truncated Paths , 2003, CAV.

[56]  Paul Ammann,et al.  A specification-based coverage metric to evaluate test sets , 1999, Proceedings 4th IEEE International Symposium on High-Assurance Systems Engineering.

[57]  Steven P. Miller,et al.  Flight Guidance System Requirements Specification , 2003 .

[58]  Boris Beizer,et al.  Software testing techniques (2. ed.) , 1990 .

[59]  Zohar Manna,et al.  Temporal verification of reactive systems - safety , 1995 .