Integrated language definition testing: enabling test-driven language development

The reliability of compilers, interpreters, and development environments for programming languages is essential for effective software development and maintenance. They are often tested only as an afterthought. Languages with a smaller scope, such as domain-specific languages, often remain untested. General-purpose testing techniques and test case generation methods fall short in providing a low-threshold solution for test-driven language development. In this paper we introduce the notion of a language-parametric testing language (LPTL) that provides a reusable, generic basis for declaratively specifying language definition tests. We integrate the syntax, semantics, and editor services of a language under test into the LPTL for writing test inputs. This paper describes the design of an LPTL and the tool support provided for it, shows use cases using examples, and describes our implementation in the form of the Spoofax testing language.

[1]  Boris Beizer,et al.  Software Testing Techniques , 1983 .

[2]  Eelco Visser,et al.  Syntax definition for language prototyping , 1997 .

[3]  Richard W. Selby.  Software Engineering: Barry W. Boehm's Lifetime Contributions to Software Development, Management, and Research , 2007 .

[4]  Kent L. Beck,et al.  Test-driven Development - by example , 2002, The Addison-Wesley signature series.

[5]  Martin Fowler,et al.  Refactoring - Improving the Design of Existing Code , 1999, Addison Wesley object technology series.

[6]  Stanley M. Sutton,et al.  Accelerating the creation of customized, language-Specific IDEs in Eclipse , 2009, OOPSLA 2009.

[7]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[8]  Terence Parr,et al.  ANTLRWorks: an ANTLR grammar development environment , 2008, Softw. Pract. Exp..

[9]  Paul Hudak,et al.  Modular domain specific languages and tools , 1998, Proceedings. Fifth International Conference on Software Reuse (Cat. No.98TB100203).

[10]  Eelco Visser,et al.  Stratego/XT 0.17. A language and toolset for program transformation , 2008, Sci. Comput. Program..

[11]  Abdulazeez S. Boujarwah,et al.  Compiler test case generation methods: a survey and assessment , 1997, Inf. Softw. Technol..

[12]  Konstantin Solomatov,et al.  Language Modularization and Composition with Projectional Language Workbenches illustrated with MPS , 2010 .

[13]  John B. Goodenough.  The Ada Compiler Validation Capability , 1981 .

[14]  Eelco Visser,et al.  Natural and Flexible Error Recovery for Generated Parsers , 2009, SLE.

[15]  Juan Carlos Augusto,et al.  Testing an Event Specification Language , 2001, SEKE.

[16]  Mark Strembeck,et al.  An approach for the systematic development of domain‐specific languages , 2009, Softw. Pract. Exp..

[17]  Andrew Meneely,et al.  Fifteen compilers in fifteen days , 2006, SIGCSE '06.

[18]  Kent Beck,et al.  Test-infected: programmers love writing tests , 2000 .

[19]  Eelco Visser,et al.  Declaratively programming the mobile web with Mobl , 2011, OOPSLA '11.

[20]  Eelco Visser,et al.  Providing rapid feedback in generated modular language environments: adding error recovery to scannerless generalized-LR parsing , 2009, OOPSLA 2009.

[21]  Eelco Visser,et al.  Concrete syntax for objects: domain-specific language embedding and assimilation without restrictions , 2004, OOPSLA '04.

[22]  Mikhail Posypkin,et al.  Survey of Compiler Testing Methods , 2005, Programming and Computer Software.

[23]  Laurence Tratt,et al.  Domain specific language implementation via compile-time meta-programming , 2008, TOPL.

[24]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[25]  Corrado Priami,et al.  Comparison of syntactic error handling in LR parsers , 1995, Softw. Pract. Exp..

[26]  Brian A. Malloy,et al.  Applying software engineering techniques to parser design: the development of a C # parser , 2002 .

[27]  Brian A. Wichmann,et al.  Pascal compiler validation , 1983 .

[28]  Ralf Lämmel,et al.  Grammar Testing , 2001, FASE.

[29]  Hui Wu,et al.  Grammar-driven generation of domain-specific language debuggers , 2008 .

[30]  Bernhard Rumpe,et al.  MontiCore: Modular Development of Textual Domain Specific Languages , 2008, TOOLS.

[31]  Eelco Visser,et al.  Pure and declarative syntax definition: paradise lost and regained , 2010, OOPSLA.

[32]  Eelco Visser,et al.  The spoofax language workbench: rules for declarative specification of languages and IDEs , 2010, OOPSLA.

[33]  Paul Hamill.  Unit Test Frameworks , 2004 .

[34]  Eelco Visser,et al.  WebDSL: A Case Study in Domain-Specific Language Engineering , 2007, GTTSE.

[35]  Eelco Visser,et al.  Interactive Disambiguation of Meta Programs with Concrete Object Syntax , 2010, SLE.

[36]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.