Grammar-based whitebox fuzzing

Whitebox fuzzing is a form of automatic dynamic test generation, based on symbolic execution and constraint solving, designed for security testing of large applications. Unfortunately, the current effectiveness of whitebox fuzzing is limited when testing applications with highly-structured inputs, such as compilers and interpreters. These applications process their inputs in stages, such as lexing, parsing and evaluation. Due to the enormous number of control paths in early processing stages, whitebox fuzzing rarely reaches parts of the application beyond those first stages. In this paper, we study how to enhance whitebox fuzzing of complex structured-input applications with a grammar-based specification of their valid inputs. We present a novel dynamic test generation algorithm where symbolic execution directly generates grammar-based constraints whose satisfiability is checked using a custom grammar-based constraint solver. We have implemented this algorithm and evaluated it on a large security-critical application, the JavaScript interpreter of Internet Explorer 7 (IE7). Results of our experiments show that grammar-based whitebox fuzzing explores deeper program paths and avoids dead-ends due to non-parsable inputs. Compared to regular whitebox fuzzing, grammar-based whitebox fuzzing increased coverage of the code generation module of the IE7 JavaScript interpreter from 53% to 81% while using three times fewer tests.

[1]  K. V. Hanford,et al.  Automatic Generation of Test Cases , 1970, IBM Syst. J..

[2]  David Coppit,et al.  yagg: an easy-to-use generator for structured test inputs , 2005, ASE.

[3]  Emin Gün Sirer,et al.  Using production grammars in software testing , 1999, DSL '99.

[4]  Helen J. Wang,et al.  Generic Application-Level Protocol Analyzer and its Language , 2007, NDSS.

[5]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[6]  Peter M. Maurer,et al.  Generating test data with enhanced context-free grammars , 1990, IEEE Software.

[7]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[8]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[9]  Ralf Lämmel,et al.  Controllable Combinatorial Coverage in Grammar-Based Testing , 2006, TestCom.

[10]  Larry L. Peterson,et al.  binpac: a yacc for writing application protocol parsers , 2006, IMC '06.

[11]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[12]  Zhendong Su,et al.  Sound and precise analysis of web applications for injection vulnerabilities , 2007, PLDI '07.

[13]  Sarfraz Khurshid,et al.  Software assurance by bounded exhaustive testing , 2004, IEEE Transactions on Software Engineering.

[14]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[15]  Brian A. Malloy,et al.  An Interpretation of Purdom's Algorithm forAutomatic Generation of Test Cases , 2001 .

[16]  Dave Aitel,et al.  The Advantages of Block - Based Protocol Analysis for Security Testing , 2002 .

[17]  Pedram Amini,et al.  Fuzzing: Brute Force Vulnerability Discovery , 2007 .

[18]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[19]  P. Purdom.  A sentence generator for testing parsers , 1972 .

[20]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[21]  Patrice Godefroid,et al.  Active property checking , 2008, EMSOFT '08.

[22]  Rupak Majumdar,et al.  Directed test generation using symbolic grammars , 2007, ASE.

[23]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[24]  Gordon V. Cormack,et al.  Scannerless NSLR(1) parsing of programming languages , 1989, PLDI '89.

[25]  Helen J. Wang,et al.  Discoverer: Automatic Protocol Reverse Engineering from Network Traces , 2007, USENIX Security Symposium.

[26]  Rupak Majumdar,et al.  Dynamic test input generation for database applications , 2007, ISSTA '07.

[27]  Bruno Legeard,et al.  A taxonomy of model-based testing , 2006 .

[28]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[29]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[30]  Darko Marinov,et al.  Automated testing of refactoring engines , 2007, ESEC-FSE '07.

[31]  Bruce McKenzie.  Generating Strings at Random from a Context Free Grammar , 1997 .

[32]  Rupak Majumdar,et al.  LATEST : Lazy Dynamic Test Input Generation , 2007 .

[33]  Frank Tip,et al.  Finding bugs in dynamic web applications , 2008, ISSTA '08.

[34]  Thomas Reps,et al.  Interconveritibility of Set Constraints and Context-Free Language Reachability , 1997, PEPM.

[35]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[36]  M. W. Shields.  An Introduction to Automata Theory , 1988 .

[37]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[38]  Sarfraz Khurshid,et al.  Korat: automated testing based on Java predicates , 2002, ISSTA '02.

[39]  Robert C. Moore.  Removing Left Recursion from Context-Free Grammars , 2000, ANLP.

[40]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.