How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection

Security attacks targeting smart contracts have been on the rise, which have led to financial loss and erosion of trust. Therefore, it is important to enable developers to discover security vulnerabilities in smart contracts before deployment. A number of static analysis tools have been developed for finding security bugs in smart contracts. However, despite the numerous bug-finding tools, there is no systematic approach to evaluate the proposed tools and gauge their effectiveness. This paper proposes SolidiFI, an automated and systematic approach for evaluating smart contracts’ static analysis tools. SolidiFI is based on injecting bugs (i.e., code defects) into all potential locations in a smart contract to introduce targeted security vulnerabilities. SolidiFI then checks the generated buggy contract using the static analysis tools, and identifies the bugs that the tools are unable to detect (false-negatives) along with identifying the bugs reported as false-positives. SolidiFI is used to evaluate six widely-used static analysis tools, namely, Oyente, Securify, Mythril, SmartCheck, Manticore and Slither, using a set of 50 contracts injected by 9369 distinct bugs. It finds several instances of bugs that are not detected by the evaluated tools despite their claims of being able to detect such bugs, and all the tools report many false positives.

[1]  Benjamin Livshits,et al.  Smart Contract Vulnerabilities: Does Anyone Care? , 2019, ArXiv.

[2]  Ye Liu,et al.  ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[3]  Chris Dannen,et al.  Introducing Ethereum and Solidity , 2017 .

[4]  Haoran Wu,et al.  Mutation Testing for Ethereum Smart Contract , 2019, ArXiv.

[5]  Massimo Bartoletti,et al.  A Survey of Attacks on Ethereum Smart Contracts (SoK) , 2017, POST.

[6]  Yoichi Hirai,et al.  Defining the Ethereum Virtual Machine for Interactive Theorem Provers , 2017, Financial Cryptography Workshops.

[7]  Sidney Amani,et al.  Towards verifying ethereum smart contract bytecode in Isabelle/HOL , 2018, CPP.

[8]  Matteo Maffei,et al.  A Semantic Framework for the Security Analysis of Ethereum smart contracts , 2018, POST.

[9]  Vitalik Buterin.  A NEXT GENERATION SMART CONTRACT & DECENTRALIZED APPLICATION PLATFORM , 2015 .

[10]  Chao Peng,et al.  SIF: A Framework for Solidity Contract Instrumentation and Analysis , 2019, 2019 26th Asia-Pacific Software Engineering Conference (APSEC).

[11]  Prateek Saxena,et al.  Making Smart Contracts Smarter , 2016, IACR Cryptol. ePrint Arch..

[12]  Chao Peng,et al.  SolAnalyser: A Framework for Analysing and Testing Smart Contracts , 2019, 2019 26th Asia-Pacific Software Engineering Conference (APSEC).

[13]  Bo Jiang,et al.  Fuse: An Architecture for Smart Contract Fuzz Testing Service , 2018, 2018 25th Asia-Pacific Software Engineering Conference (APSEC).

[14]  Chris Dannen,et al.  Introducing Ethereum and Solidity: Foundations of Cryptocurrency and Blockchain Programming for Beginners , 2017 .

[15]  Thorsten Holz,et al.  EvilCoder: automated bug insertion , 2016, ACSAC.

[16]  Alex Groce,et al.  Slither: A Static Analysis Framework for Smart Contracts , 2019, 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB).

[17]  Sergei Tikhomirov,et al.  SmartCheck: Static Analysis of Ethereum Smart Contracts , 2018, 2018 IEEE/ACM 1st International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB).

[18]  Nikhil Swamy,et al.  Formal Verification of Smart Contracts: Short Paper , 2016, PLAS@CCS.

[19]  Petar Tsankov,et al.  Securify: Practical Security Analysis of Smart Contracts , 2018, CCS.

[20]  Premkumar T. Devanbu,et al.  To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools , 2012, 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering.

[21]  Emina Torlak,et al.  Precise Attack Synthesis for Smart Contracts , 2019, ArXiv.

[22]  Nishant Rodrigues,et al.  KEVM: A Complete Semantics of the Ethereum Virtual Machine , 2017 .

[23]  Denys Poshyvanyk,et al.  Discovering Flaws in Security-Focused Static Analysis Tools for Android using Systematic Mutation , 2018, USENIX Security Symposium.

[24]  Vincent Gramoli,et al.  Vandal: A Scalable Security Analysis Framework for Smart Contracts , 2018, ArXiv.

[25]  Rui Abreu,et al.  Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[26]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[27]  William K. Robertson,et al.  LAVA: Large-Scale Automated Vulnerability Addition , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[28]  Christian Rossow,et al.  teEther: Gnawing at Ethereum to Automatically Exploit Smart Contracts , 2018, USENIX Security Symposium.

[29]  Christopher D. Clack,et al.  Smart Contract Templates: foundations, design landscape and research directions , 2016, ArXiv.

[30]  Haoran Wu,et al.  Towards Generating Cost-Effective Test-Suite for Ethereum Smart Contract , 2019, 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[31]  Alex Groce,et al.  Manticore: A User-Friendly Symbolic Execution Framework for Binaries and Smart Contracts , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[32]  Ali Dehghantanha,et al.  Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains , 2018, CASCON.