A Generic and Automatic Test Strategy for Compiler Testing

Domain-specific Languages (DSLs) are languages specifically tailored for an application or expert domain. These can be implemented as compilers, which check the correctness of an input program and translates it to a target language. Manual testing of compilers is a time consuming and labor intensive task. This motivates the development of approaches to facilitate the quality assurance process. In this thesis we present an automatic and generic test strategy for the generation of test cases for Spoofax developed compilers. We use a program generator to generate large syntactically correct programs from Syntax Definition Formalism (SDF) grammars. Additionally, we improve the program generator with an expansion of our generation algorithm to use Name Binding Language (NaBL) modules to generate partial name correct programs. We also provide a DSL to define error fixes that are used to attempt the repair of static semantic errors reported after compilation. After program generation we use a partial oracle to automatically detect failures during the invocation of the compiler. Finally, we provide a heuristic to reduce the size of generated programs, whilst preserving their failure inducing behavior. This test strategy was used to generate test cases forWebDSL, a DSL targeting the domain of developing dynamic web applications with a rich data model. The generated test cases unveiled eleven unique faults in the analysis phase of compilation. These were reported together with the programs reduced by our program shrinking heuristic and they were positively received by the WebDSL development team.

[1]  Koen Claessen,et al.  Testing an optimising compiler by generating random lambda terms , 2011, AST '11.

[2]  Mikhail Posypkin,et al.  Survey of Compiler Testing Methods , 2005, Programming and Computer Software.

[3]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[4]  John Hughes,et al.  Software Testing with QuickCheck , 2009, CEFP.

[5]  Andreas Zeller,et al.  Simplifying failure-inducing input , 2000, ISSTA '00.

[6]  Peter M. Maurer,et al.  Generating test data with enhanced context-free grammars , 1990, IEEE Software.

[7]  Eelco Visser,et al.  Program Transformation with Scoped Dynamic Rewrite Rules , 2006, Fundam. Informaticae.

[8]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[9]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[10]  William E. Howden,et al.  Tutorial : software testing & validation techniques , 1981 .

[11]  Paul A. Strooper,et al.  Grammar‐based test generation with YouGen , 2011, Softw. Pract. Exp..

[12]  K. V. Hanford,et al.  Automatic Generation of Test Cases , 1970, IBM Syst. J..

[13]  Eelco Visser,et al.  The spoofax language workbench: rules for declarative specification of languages and IDEs , 2010, OOPSLA.

[14]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[15]  Eelco Visser,et al.  WebDSL: A Case Study in Domain-Specific Language Engineering , 2007, GTTSE.

[16]  Casey Klein,et al.  Randomized Testing in PLT Redex , 2009 .

[17]  Eelco Visser,et al.  Syntax definition for language prototyping , 1997 .

[18]  Eelco Visser,et al.  Declaratively defining domain-specific language debuggers , 2011, GPCE '11.

[19]  D. Parigot,et al.  Two-dimensional Approximation Coverage , 2000 .

[20]  Paul Walton Purdom,et al.  A sentence generator for testing parsers , 1972 .

[21]  Ralf Lämmel,et al.  Controllable Combinatorial Coverage in Grammar-Based Testing , 2006, TestCom.

[22]  Sam Tobin-Hochstadt,et al.  Run your research: on the effectiveness of lightweight mechanization , 2012, POPL '12.

[23]  Abdulazeez S. Boujarwah,et al.  Compiler test case generation methods: a survey and assessment , 1997, Inf. Softw. Technol..

[24]  Xuejun Yang,et al.  Test-case reduction for C compiler bugs , 2012, PLDI.

[25]  Mariusz A. Fecko,et al.  Testing of Communicating Systems: 18th IFIP TC 6/WG 6.1 International Conference, TestCom 2006, New York, NY, USA, May 16-18, 2006, Proceedings (Lecture Notes in Computer Science) , 2006 .

[26]  Zhendong Su,et al.  HDD: hierarchical delta debugging , 2006, ICSE.

[27]  Emin Gün Sirer,et al.  Using production grammars in software testing , 1999, DSL '99.

[28]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[29]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[30]  Anneke Kleppe,et al.  MDA explained - the Model Driven Architecture: practice and promise , 2003, Addison Wesley object technology series.

[31]  René Just,et al.  Automating software tests with partial oracles in integrated environments , 2010, AST '10.

[32]  B. B. Pat-El.  Automated test generation for microsoft DSL tools , 2010 .

[33]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools (2nd Edition) , 2006 .

[34]  M Mernik,et al.  When and how to develop domain-specific languages , 2005, CSUR.

[35]  Eelco Visser,et al.  Separation of Concerns and Linguistic Integration in WebDSL , 2010, IEEE Software.

[36]  Eelco Visser,et al.  Integrated language definition testing: enabling test-driven language development , 2011, OOPSLA '11.

[37]  Bas Basten,et al.  Ambiguity Detection for Programming Language Grammars , 2011 .

[38]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[39]  Eelco Visser,et al.  Declarative Name Binding and Scope Rules , 2012, SLE.

[40]  Alfred V. Aho,et al.  Compilers: Principles, Techniques, and Tools , 1986, Addison-Wesley series in computer science / World student series edition.

[41]  João Saraiva,et al.  Generative and Transformational Techniques in Software Engineering II , 2007, Lecture Notes in Computer Science.

[42]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.