An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software

The term grammar-based software describes software whose input can be specified by a context-free grammar. This grammar may occur explicitly in the software, in the form of an input specification to a parser generator, or implicitly, in the form of a hand-written parser, or other input-verification routines. Grammar-based software includes not only programming language compilers, but also tools for program analysis, reverse engineering, software metrics and documentation generation. Such tools often play a crucial role in automated software development, and ensuring their completeness and correctness is a vital prerequisite for their us.In this paper we propose a strategy for the construction of test suites for grammar based software, and illustrate this strategy using the ISO CPP grammar. We use the concept of rule coverage as a pivot for the reduction of implementation-based and specification-based test suites, and demonstrate a significant decrease in the size of these suites. To demonstrate the validity of the approach, we use the reduced test suite to analyze three grammar-based tools for CPP++. We compare the effectiveness of the reduced test suite with the original suite in terms of code coverage and fault detection.

[1]  Paul Walton Purdom,et al.  A sentence generator for testing parsers , 1972 .

[2]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[3]  Carlo Ghezzi,et al.  Compiler testing using a sentence generator , 1980, Softw. Pract. Exp..

[4]  Ippolito Spadafora,et al.  An Automatic Generator for Compiler Testing , 1982, IEEE Transactions on Software Engineering.

[5]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[6]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[7]  Joseph Robert Horgan,et al.  Effect of Test Set Minimization on Fault Detection Effectiveness , 1995, 1995 17th International Conference on Software Engineering.

[8]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[9]  Chris Verhoef,et al.  Current parsing techniques in software renovation considered harmful , 1998, Proceedings. 6th International Workshop on Program Comprehension. IWPC'98 (Cat. No.98TB100242).

[10]  D. Parigot,et al.  Two-dimensional Approximation Coverage , 2000 .

[11]  Brian A. Malloy,et al.  An Interpretation of Purdom's Algorithm forAutomatic Generation of Test Cases , 2001 .

[12]  Ralf Lämmel,et al.  Grammar Testing , 2001, FASE.

[13]  J.F. Power.  Program annotation in XML : a parser-based approach , 2002 .

[14]  Brian A. Malloy,et al.  Testing C++ Compilers for ISO Language Conformance , 2002 .

[15]  Brian A. Malloy,et al.  Program annotation in XML: a parse-tree based approach , 2002, Ninth Working Conference on Reverse Engineering, 2002. Proceedings..

[16]  Wolfgang Schröder-Preikschat,et al.  AspectC++: an aspect-oriented extension to the C++ programming language , 2002 .

[17]  Brian A. Malloy,et al.  Programmer's toolchest: testing C++ compilers for ISO language conformance , 2002 .

[18]  Brian A. Malloy,et al.  Applying software engineering techniques to parser design: the development of a C # parser , 2002 .

[19]  Susan Elliott Sim,et al.  On using a benchmark to evaluate C++ extractors , 2002, Proceedings 10th International Workshop on Program Comprehension.

[20]  Brian A. Malloy,et al.  Progression Toward Conformance for C++ Language Compilers , 2003 .

[21]  Mary Jean Harrold,et al.  Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage , 2003, IEEE Trans. Software Eng..

[22]  Brian A. Malloy,et al.  Decorating tokens to facilitate recognition of ambiguous language constructs , 2003, Softw. Pract. Exp..

[23]  Mark Hennessy,et al.  gccXfront: exploiting gcc as a front end for program comprehension tools via XML/XSLT , 2003, 11th IEEE International Workshop on Program Comprehension, 2003..

[24]  Brian A. Malloy,et al.  A metrics suite for grammar-based software , 2004, J. Softw. Maintenance Res. Pract..

[25]  M. Heimdahl,et al.  Test-suite reduction for model based tests: effects on test quality and implications for testing , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[26]  Brian A. Malloy,et al.  A metrics suite for grammar-based software: Research Articles , 2004 .

[27]  Ralf Lämmel,et al.  Towards an engineering discipline for GRAMMARWARE Draft as of August 17 , 2003 , 2003 .

[28]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..