Synthesizing program input grammars

We present an algorithm for synthesizing a context-free grammar encoding the language of valid program inputs from a set of input examples and blackbox access to the program. Our algorithm addresses shortcomings of existing grammar inference algorithms, which both severely overgeneralize and are prohibitively slow. Our implementation, GLADE, leverages the grammar synthesized by our algorithm to fuzz test programs with structured inputs. We show that GLADE substantially increases the incremental coverage on valid inputs compared to two baseline fuzzers.

[1]  Ray J. Solomonoff,et al.  A new method for discovering the grammars of phrase structure languages , 1959, IFIP Congress.

[2]  Richard L. Sauder,et al.  A general test data generator for COBOL , 1962, AIEE-IRE '62 (Spring).

[3]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[4]  Paul Walton Purdom,et al.  A sentence generator for testing parsers , 1972 .

[5]  Kathleen Knobe,et al.  A Method for Inferring Context-free Grammars , 1976, Inf. Control..

[6]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[7]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[8]  Pedro García,et al.  IDENTIFYING REGULAR LANGUAGES IN POLYNOMIAL TIME , 1993 .

[9]  Andreas Stolcke,et al.  Bayesian learning of probabilistic language models , 1994 .

[10]  Lillian Lee,et al.  Learning of Context-Free Languages: A Survey of the Literature , 1996 .

[11]  Abdulazeez S. Boujarwah,et al.  Compiler test case generation methods: a survey and assessment , 1997, Inf. Softw. Technol..

[12]  Barton P. Miller,et al.  An empirical study of the robustness of Windows NT applications using random testing , 2000 .

[13]  M. Rinard.  Acceptability-oriented computing , 2003, OOPSLA '03.

[14]  Mahesh Viswanathan,et al.  Learning to Verify Safety Properties , 2004, ICFEM.

[15]  Christian Lindig,et al.  Random testing of C calling conventions , 2005, AADEBUG'05.

[16]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[17]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[18]  H. Ishizaka.  Polynomial Time Learnability of Simple Deterministic Languages , 1990, Machine Learning.

[19]  Zhendong Su,et al.  The essence of command injection attacks in web applications , 2006, POPL '06.

[20]  Barton P. Miller,et al.  An empirical study of the robustness of MacOS applications using random testing , 2006, RT '06.

[21]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[22]  Rupak Majumdar,et al.  Directed test generation using symbolic grammars , 2007, ESEC-FSE companion '07.

[23]  Pedram Amini,et al.  Fuzzing: Brute Force Vulnerability Discovery , 2007 .

[24]  Martin C. Rinard,et al.  Living in the comfort zone , 2007, OOPSLA.

[25]  Zhenkai Liang,et al.  Polyglot: automatic extraction of protocol message format using dynamic binary analysis , 2007, CCS '07.

[26]  Xiangyu Zhang,et al.  Deriving input syntactic structure from execution , 2008, SIGSOFT '08/FSE-16.

[27]  Frank Tip,et al.  Finding bugs in dynamic web applications , 2008, ISSTA '08.

[28]  Christopher Krügel,et al.  Automatic Network Protocol Analysis , 2008, NDSS.

[29]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[30]  Juha Röning,et al.  Experiences with Model Inference Assisted Fuzzing , 2008, WOOT.

[31]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[32]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[33]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[34]  Martin C. Rinard,et al.  Taint-based directed whitebox fuzzing , 2009, 2009 IEEE 31st International Conference on Software Engineering.

[35]  Rafael C. Carrasco,et al.  Inducing Probabilistic Grammars by Bayesian Model Merging , 2009 .

[36]  Benedikt Bollig,et al.  libalf: The Automata Learning Framework , 2010, CAV.

[37]  Colin de la Higuera,et al.  Grammatical Inference: Learning Automata and Grammars , 2010 .

[38]  Guofei Gu,et al.  TaintScope: A Checksum-Aware Directed Fuzzing Tool for Automatic Software Vulnerability Detection , 2010, 2010 IEEE Symposium on Security and Privacy.

[39]  Ling Huang,et al.  Predicting Execution Time of Computer Programs Using Sparse Polynomial Regression , 2010, NIPS.

[40]  Xiangyu Zhang,et al.  Reverse Engineering Input Syntactic Structure from Program Execution and Its Applications , 2010, IEEE Transactions on Software Engineering.

[41]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[42]  Dawn Xiaodong Song,et al.  MACE: Model-inference-Assisted Concolic Exploration for Protocol and Vulnerability Discovery , 2011, USENIX Security Symposium.

[43]  Sumit Gulwani,et al.  Spreadsheet table transformations from examples , 2011, PLDI '11.

[44]  Sumit Gulwani,et al.  Automating string processing in spreadsheets using input-output examples , 2011, POPL '11.

[45]  Sumit Gulwani,et al.  Synthesizing Number Transformations from Input-Output Examples , 2012, CAV.

[46]  Andreas Zeller,et al.  Fuzzing with Code Fragments , 2012, USENIX Security Symposium.

[47]  Hongseok Yang,et al.  Abstractions from tests , 2012, POPL '12.

[48]  Zvonimir Rakamaric,et al.  Symbolic Learning of Component Interfaces , 2012, SAS.

[49]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[50]  Sumit Gulwani,et al.  Recursive Program Synthesis , 2013, CAV.

[51]  Domagoj Babic,et al.  Sigma*: symbolic learning of input-output specifications , 2013, POPL.

[52]  Gilad Bracha,et al.  The Java Virtual Machine Specification, Java SE 8 Edition , 2013 .

[53]  George C. Necula,et al.  Guided GUI testing of android apps with minimal restart and approximate learning , 2013, OOPSLA.

[54]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[55]  Sumit Gulwani,et al.  Test-driven synthesis , 2014, PLDI.

[56]  Sorin Lerner,et al.  Interactive parser synthesis by example , 2015, PLDI.

[57]  Sumit Gulwani,et al.  FlashMeta: a framework for inductive program synthesis , 2015, OOPSLA.

[58]  Isil Dillig,et al.  Synthesizing data structure transformations from input-output examples , 2015, PLDI.

[59]  Andreas Zeller,et al.  Mining input grammars from dynamic taints , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).