Using production grammars in software testing

Extensible typesafe systems, such as Java, rely critically on a large and complex software base for their overall protection and integrity, and are therefore difficult to test and verify. Traditional testing techniques, such as manual test generation and formal verification, are too time consuming, expensive, and imprecise, or work only on abstract models of the implementation and are too simplistic. Consequently, commercial virtual machines deployed so far have exhibited numerous bugs and security holes. In this paper, we discuss our experience with using production grammars in testing large, complex and safety-critical software systems. Specifically, we describe lava, a domain specific language we have developed for specifying production grammars, and relate our experience with using lava to generate effective test suites for the Java virtual machine. We demonstrate the effectiveness of production grammars in generating complex test cases that can, when combined with comparative and variant testing techniques, achieve high code and value coverage. We also describe an extension to production grammars that enables concurrent generation of certificates for test cases. A certificate is a behavioral description that specifies the intended outcome of the generated test case, and therefore acts as an oracle by which the correctness of the tested system can be evaluated in isolation. We report the results of applying these testing techniques to commercial Java implementations. We conclude that the use of production grammars in combination with other automated testing techniques is a powerful and effective method for testing software systems, and is enabled by a special purpose language for specifying extended production grammars.

[1]  Jonathan A. Bauer,et al.  Test Plan Generation Using Formal Grammars , 1979, ICSE.

[2]  Carlo Ghezzi,et al.  Compiler testing using a sentence generator , 1980, Softw. Pract. Exp..

[3]  Przemyslaw Prusinkiewicz,et al.  Development models of herbaceous plants for computer imagery purposes , 1988, SIGGRAPH.

[4]  Alfred V. Aho,et al.  The awk programming language , 1988 .

[5]  Jean-Pierre Jouannaud,et al.  Rewrite Systems , 1991, Handbook of Theoretical Computer Science, Volume B: Formal Models and Sematics.

[6]  Nachum Dershowitz,et al.  A Taste of Rewrite Systems , 1993, Functional Programming, Concurrency, Simulation and Automated Reasoning.

[7]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[8]  Sandro Morasca,et al.  Generating test cases for real-time systems from logic specifications , 1995, TOCS.

[9]  Robert Wahbe,et al.  Efficient and language-independent mobile programs , 1996, PLDI '96.

[10]  Frank Yellin,et al.  The Java Virtual Machine Specification , 1996 .

[11]  Gary McGraw,et al.  Java Security , 1996 .

[12]  Charles Consel,et al.  Harissa: A Flexible and Efficient Java Environment Mixing Bytecode and Compiled Code , 1997, COOTS.

[13]  Gary McGraw,et al.  Java security - hostile applets, holes and antidotes: what every netscape and internet explorer user needs to know , 1997 .

[14]  Troy Downing,et al.  Java Virtual Machine , 1997 .

[15]  Sophia Drossopoulou,et al.  Java is Type Safe - Probably , 1997, ECOOP.

[16]  George C. Necula,et al.  Proof-carrying code , 1997, POPL '97.

[17]  John H. Hartman,et al.  Toba: Java for Applications - A Way Ahead of Time (WAT) Compiler , 1997, COOTS.

[18]  Dan S. Wallach,et al.  Java security: Web browsers and beyond , 1997 .

[19]  Gary McGraw,et al.  Java security: hostile applets, holes&antidotes , 1997 .

[20]  Stephen N. Freund,et al.  A type system for object initialization in the Java bytecode language , 1998, OOPSLA '98.

[21]  M. Abadi,et al.  A type system for Java bytecode subroutines , 1998, POPL '98.

[22]  George C. Necula,et al.  The design and implementation of a certifying compiler , 1998, PLDI.

[23]  Emin Gün Sirer,et al.  Distributed virtual machines: a system architecture for network computing , 1998, ACM SIGOPS European Workshop.

[24]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[25]  Angelo Gargantini,et al.  Using model checking to generate tests from requirements specifications , 1999, ESEC/FSE-7.

[26]  Martín Abadi,et al.  A type system for Java bytecode subroutines , 1999, TOPL.

[27]  Don Syme,et al.  Proving Java Type Soundness , 1999, Formal Syntax and Semantics of Java.

[28]  Sarfraz Khurshid,et al.  Is the Java Type System Sound? , 1999, Theory Pract. Object Syst..