Testing an Optimising Compiler by Generating Random Lambda Terms

This thesis tries to improve on the relatively uncommon practice of random testing of compilers. Random testing of compilers is difficult and not widespread for two reasons. First, it is hard to come up with a generator of valid test data for compilers, that is a generator of programs. And secondly, it is difficult to provide a specification, or test oracle, that decides what should be the correct behaviour of a compiler. This work addresses both of these problems. Existing random compiler test tools do not use a structured way of generating well-typed programs, which is a often a requirement to perform comprehensive testing of a compiler. This thesis proposes such a method based on a formal calculus. To address the second problem, this thesis proposes using two variants of differential testing, which allows for detecting bugs even when a very limited partial specification of the tested compiler is available. This setup is evaluated practically by performing effective testing of a real compiler.

[1]  K. V. Hanford,et al.  Automatic Generation of Test Cases , 1970, IBM Syst. J..

[2]  Roy Dyckhoff,et al.  Contraction-free sequent calculi for intuitionistic logic , 1992, Journal of Symbolic Logic.

[3]  Simon L. Peyton Jones,et al.  A short cut to deforestation , 1993, FPCA '93.

[4]  Simon L. Peyton Jones,et al.  Compiling Haskell by Program Transformation: A Report from the Trenches , 1996, ESOP.

[5]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[6]  Simon L. Peyton Jones,et al.  A semantics for imprecise exceptions , 1999, PLDI '99.

[7]  K. Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP '00.

[8]  Jerzy Tyszkiewicz,et al.  Statistical properties of simple types , 2000, Mathematical Structures in Computer Science.

[9]  David Sands,et al.  Possibilities and limitations of call-by-need space improvement , 2001, ICFP '01.

[10]  Simon Peyton Jones,et al.  Playing by the rules: rewriting as a practical optimisation technique in GHC , 2001 .

[11]  Koen Claessen,et al.  Testing monadic code with QuickCheck , 2002, Haskell '02.

[12]  Benjamin C. Pierce,et al.  Types and programming languages: the next generation , 2003, 18th Annual IEEE Symposium of Logic in Computer Science, 2003. Proceedings..

[13]  G. Tassey.  The economic impacts of inadequate infrastructure for software testing , 2002 .

[14]  Andreas Zeller,et al.  Simplifying and Isolating Failure-Inducing Input , 2002, IEEE Trans. Software Eng..

[15]  Peyton Jones,et al.  Haskell 98 language and libraries : the revised report , 2003 .

[16]  Nils Anders Danielsson,et al.  Chasing Bottoms: A Case Study in Program Verification in the Presence of Partial and Infinite Values , 2004, MPC.

[17]  Christian Lindig,et al.  Random testing of C calling conventions , 2005, AADEBUG'05.

[18]  Jue Wang.  Generating Random Lambda Calculus Terms , 2005 .

[19]  John Hughes,et al.  QuickCheck Testing for Fun and Profit , 2007, PADL.

[20]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[21]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[22]  Simon L. Peyton Jones,et al.  Runtime support for multicore Haskell , 2009, ICFP.

[23]  Johan Jeuring,et al.  Enumerating Well-Typed Terms Generically , 2009, AAIP.

[24]  Dimitrios Vytiniotis,et al.  Functional pearl: every bit counts , 2010, ICFP '10.

[25]  Simon J. Thompson,et al.  Quickchecking refactoring tools , 2010, Erlang '10.

[26]  Simon Marlow,et al.  Haskell 2010 Language Report , 2010 .

[27]  Jonas Almström-Duregård.  AGATA - Random generation of test data , 2010 .

[28]  Koen Claessen,et al.  Testing an optimising compiler by generating random lambda terms , 2011, AST '11.

[29]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[30]  Danièle Gardy,et al.  Lambda-terms of Bounded Unary Height , 2011, ANALCO.

[31]  Sam Tobin-Hochstadt,et al.  Run your research: on the effectiveness of lightweight mechanization , 2012, POPL '12.

[32]  Robert Bruce Findler,et al.  The Racket virtual machine and randomized testing , 2012, High. Order Symb. Comput..