Learning to Fuzz from Symbolic Execution with Application to Smart Contracts

Fuzzing and symbolic execution are two complementary techniques for discovering software vulnerabilities. Fuzzing is fast and scalable, but can be ineffective when it fails to randomly select the right inputs. Symbolic execution is thorough but slow and often does not scale to deep program paths with complex path conditions. In this work, we propose to learn an effective and fast fuzzer from symbolic execution, by phrasing the learning task in the framework of imitation learning. During learning, a symbolic execution expert generates a large number of quality inputs improving coverage on thousands of programs. Then, a fuzzing policy, represented with a suitable architecture of neural networks, is trained on the generated dataset. The learned policy can then be used to fuzz new programs. We instantiate our approach to the problem of fuzzing smart contracts, a domain where contracts often implement similar functionality (facilitating learning) and security is of utmost importance. We present an end-to-end system, ILF (for Imitation Learning based Fuzzer), and an extensive evaluation over >18K contracts. Our results show that ILF is effective: (i) it is fast, generating 148 transactions per second, (ii) it outperforms existing fuzzers (e.g., achieving 33% more coverage), and (iii) it detects more vulnerabilities than existing fuzzing and symbolic execution tools for Ethereum.

[1]  Mislav Balunovic,et al.  Learning to Solve SMT Formulas , 2018, NeurIPS.

[2]  Prateek Saxena,et al.  Finding The Greedy, Prodigal, and Suicidal Contracts at Scale , 2018, ACSAC.

[3]  Marc Brockschmidt,et al.  Learning to Represent Programs with Graphs , 2017, ICLR.

[4]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[5]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[6]  Sidney Amani,et al.  Towards verifying ethereum smart contract bytecode in Isabelle/HOL , 2018, CPP.

[7]  Dawn Xiaodong Song,et al.  Recognizing Functions in Binaries with Neural Networks , 2015, USENIX Security Symposium.

[8]  Rishabh Singh,et al.  Learn&Fuzz: Machine learning for input fuzzing , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[9]  Mathias Payer,et al.  T-Fuzz: Fuzzing by Program Transformation , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[10]  Nikhil Swamy,et al.  Formal Verification of Smart Contracts: Short Paper , 2016, PLAS@CCS.

[11]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[12]  Christopher Krügel,et al.  Driller: Augmenting Fuzzing Through Selective Symbolic Execution , 2016, NDSS.

[13]  Chris Cummins,et al.  Compiler fuzzing through deep learning , 2018, ISSTA.

[14]  Yoichi Hirai,et al.  Defining the Ethereum Virtual Machine for Interactive Theorem Provers , 2017, Financial Cryptography Workshops.

[15]  Claudia Eckert,et al.  Empowering convolutional networks for malware classification and analysis , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[16]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[17]  Shweta Shinde,et al.  Neuro-Symbolic Execution: Augmenting Symbolic Execution with Neural Constraints , 2019, NDSS.

[18]  Petar Tsankov,et al.  Securify: Practical Security Analysis of Smart Contracts , 2018, CCS.

[19]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[20]  Herbert Bos,et al.  VUzzer: Application-aware Evolutionary Fuzzing , 2017, NDSS.

[21]  Prateek Saxena,et al.  Making Smart Contracts Smarter , 2016, IACR Cryptol. ePrint Arch..

[22]  Yi Zhang,et al.  KEVM: A Complete Formal Semantics of the Ethereum Virtual Machine , 2018, 2018 IEEE 31st Computer Security Foundations Symposium (CSF).

[23]  Ye Liu,et al.  ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[24]  Patrice Godefroid,et al.  Automated Whitebox Fuzz Testing , 2008, NDSS.

[25]  Alan Mislove,et al.  Analyzing Ethereum's Contract Topology , 2018, Internet Measurement Conference.

[26]  Le Song,et al.  Learning Loop Invariants for Program Verification , 2018, NeurIPS.

[27]  Junfeng Yang,et al.  NEUZZ: Efficient Fuzzing with Neural Program Learning , 2018, ArXiv.

[28]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[29]  Sukrit Kalra,et al.  ZEUS: Analyzing Safety of Smart Contracts , 2018, NDSS.

[30]  Zhendong Su,et al.  Compiler validation via equivalence modulo inputs , 2014, PLDI.

[31]  Salvatore J. Stolfo,et al.  Data mining methods for detection of new malicious executables , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[32]  Abhik Roychoudhury,et al.  Coverage-Based Greybox Fuzzing as Markov Chain , 2017, IEEE Trans. Software Eng..

[33]  Yang Liu,et al.  Steelix: program-state based binary fuzzing , 2017, ESEC/SIGSOFT FSE.

[34]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[35]  Mayur Naik,et al.  Effective Program Debloating via Reinforcement Learning , 2018, CCS.

[36]  Yang Liu,et al.  Skyfire: Data-Driven Seed Generation for Fuzzing , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[37]  Christian Rossow,et al.  teEther: Gnawing at Ethereum to Automatically Exploit Smart Contracts , 2018, USENIX Security Symposium.

[38]  Yannis Smaragdakis,et al.  MadMax: surviving out-of-gas conditions in Ethereum smart contracts , 2018, Proc. ACM Program. Lang..

[39]  Ittai Abraham,et al.  Online detection of effectively callback free objects with applications to smart contracts , 2017, Proc. ACM Program. Lang..

[40]  Yi Zhou,et al.  Erays: Reverse Engineering Ethereum's Opaque Smart Contracts , 2018, USENIX Security Symposium.

[41]  Petar Tsankov,et al.  Debin: Predicting Debug Information in Stripped Binaries , 2018, CCS.

[42]  Andreas Zeller,et al.  Mining input grammars from dynamic taints , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[43]  Radu State,et al.  Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts , 2018, ACSAC.

[44]  Markus Püschel,et al.  Fast Numerical Program Analysis with Reinforcement Learning , 2018, CAV.

[45]  Muhammad Torabi Dashti,et al.  SECFUZZ: Fuzz-testing security protocols , 2012, 2012 7th International Workshop on Automation of Software Test (AST).

[46]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[47]  Daniel Davis Wood,et al.  ETHEREUM: A SECURE DECENTRALISED GENERALISED TRANSACTION LEDGER , 2014 .

[48]  Alexander Aiken,et al.  Synthesizing program input grammars , 2016, PLDI.

[49]  Grigore Rosu,et al.  An overview of the K semantic framework , 2010, J. Log. Algebraic Methods Program..

[50]  Dean Pomerleau,et al.  ALVINN, an autonomous land vehicle in a neural network , 2015 .

[51]  Meng Xu,et al.  QSYM : A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing , 2018, USENIX Security Symposium.

[52]  David Brumley,et al.  BYTEWEIGHT: Learning to Recognize Functions in Binary Code , 2014, USENIX Security Symposium.

[53]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[54]  M. Schuyler Security alert , 1996 .

[55]  Hao Chen,et al.  Angora: Efficient Fuzzing by Principled Search , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[56]  Dimitar Dimitrov,et al.  VerX: Safety Verification of Smart Contracts , 2020, 2020 IEEE Symposium on Security and Privacy (SP).

[57]  Geoffrey J. Gordon,et al.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning , 2010, AISTATS.

[58]  Andreas Zeller,et al.  Fuzzing with Code Fragments , 2012, USENIX Security Symposium.

[59]  Pieter Abbeel,et al.  Autonomous Helicopter Aerobatics through Apprenticeship Learning , 2010, Int. J. Robotics Res..

[60]  Jack W. Stokes,et al.  Malware classification with LSTM and GRU language models and a character-level CNN , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).