DeepFuzz: Automatic Generation of Syntax Valid C Programs for Fuzz Testing

Compilers are among the most fundamental programming tools for building software. However, production compilers remain buggy. Fuzz testing is often leveraged with newlygenerated, or mutated inputs in order to find new bugs or security vulnerabilities. In this paper, we propose a grammarbased fuzzing tool called DEEPFUZZ. Based on a generative Sequence-to-Sequence model, DEEPFUZZ automatically and continuously generates well-formed C programs. We use this set of new C programs to fuzz off-the-shelf C compilers, e.g., GCC and Clang/LLVM. We present a detailed case study to analyze the success rate and coverage improvement of the generated C programs for fuzz testing. We analyze the performance of DEEPFUZZ with three types of sampling methods as well as three types of generation strategies. Consequently, DEEPFUZZ improved the testing efficacy in regards to the line, function, and branch coverage. In our preliminary study, we found and reported 8 bugs of GCC, all of which are actively being addressed by developers.

[1]  Meir Kalech,et al.  Data-Augmented Software Diagnosis , 2016, DX.

[2]  H. Cleve,et al.  Locating causes of program failures , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[3]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[4]  Jared Roesch,et al.  Language fuzzing using constraint logic programming , 2014, ASE.

[5]  Anh Tuan Nguyen,et al.  A statistical semantic language model for source code , 2013, ESEC/FSE 2013.

[6]  C. A. R. Hoare,et al.  The verifying compiler: A grand challenge for computing research , 2003, JACM.

[7]  Pedro M. Domingos,et al.  Learning Tractable Probabilistic Models for Fault Localization , 2015, AAAI.

[8]  Alexander M. Rush,et al.  OpenNMT: Open-Source Toolkit for Neural Machine Translation , 2017, ACL.

[9]  Charles A. Sutton,et al.  Mining source code repositories at massive scale using language modeling , 2013, 2013 10th Working Conference on Mining Software Repositories (MSR).

[10]  Zhendong Su,et al.  Finding deep compiler bugs via guided stochastic program mutation , 2015, OOPSLA.

[11]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[12]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[13]  Chris Cummins,et al.  Compiler fuzzing through deep learning , 2018, ISSTA.

[14]  George C. Necula,et al.  Translation validation for an optimizing compiler , 2000, PLDI '00.

[15]  Xavier Leroy,et al.  CompCert - A Formally Verified Optimizing Compiler , 2016 .

[16]  Rishabh Singh,et al.  Learn&Fuzz: Machine learning for input fuzzing , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[17]  Xing Shi,et al.  Does String-Based Neural MT Learn Source Syntax? , 2016, EMNLP.

[18]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[19]  Xavier Leroy,et al.  Coinductive big-step operational semantics , 2006, Inf. Comput..

[20]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[21]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[22]  Premkumar T. Devanbu,et al.  On the naturalness of software , 2016, Commun. ACM.

[23]  Zhendong Su,et al.  Toward understanding compiler bugs in GCC and LLVM , 2016, ISSTA.

[24]  Carlos Urias Munoz,et al.  Automatic Generation of Random Self-Checking Test Cases , 1983, IBM Syst. J..

[25]  Alastair F. Donaldson,et al.  Many-core compiler fuzzing , 2015, PLDI.

[26]  Pushmeet Kohli,et al.  RobustFill: Neural Program Learning under Noisy I/O , 2017, ICML.

[27]  Geoffrey E. Hinton,et al.  Generating Text with Recurrent Neural Networks , 2011, ICML.

[28]  Xi Victoria Lin.  Program Synthesis from Natural Language Using Recurrent Neural Networks , 2017 .

[29]  Rishabh Singh,et al.  Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks , 2016, ArXiv.

[30]  Meir Kalech,et al.  Using Model-Based Diagnosis to Improve Software Testing , 2014, AAAI.