Mutation Testing of Smart Contracts at Scale

It is crucial that smart contracts are tested thoroughly due to their immutable nature. Even small bugs in smart contracts can lead to huge monetary losses. However, testing is not enough; it is also important to ensure the quality and completeness of the tests. There are already several approaches that tackle this challenge with mutation testing, but their effectiveness is questionable since they only considered small contract samples. Hence, we evaluate the quality of smart contract mutation testing at scale. We choose the most promising of the existing (smart contract specific) mutation operators, analyse their effectiveness in terms of killability and highlight severe vulnerabilities that can be injected with the mutations. Moreover, we improve the existing mutation methods by introducing a novel killing condition that is able to detect a deviation in the gas consumption, i.e., in the monetary value that is required to perform transactions. This paper has a replication package at this https URL

[1]  Mark Harman,et al.  Using program slicing to assist in the detection of equivalent mutants , 1999, Softw. Test. Verification Reliab..

[2]  Pieter H. Hartel,et al.  Rethinking Blockchain Security: Position Paper , 2018, 2018 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData).

[3]  Ruihua Nie,et al.  Basis Path Coverage Criteria for Smart Contract Application Testing , 2019, 2019 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC).

[4]  A. Jefferson Offutt,et al.  Using compiler optimization techniques to detect equivalent mutants , 1994, Softw. Test. Verification Reliab..

[5]  Andy Zaidman,et al.  A systematic literature review of how mutation testing supports quality assurance processes , 2018, Softw. Test. Verification Reliab..

[6]  Fathiyeh Faghih,et al.  Testing Smart Contracts Gets Smarter , 2019, 2020 10th International Conference on Computer and Knowledge Engineering (ICCKE).

[7]  Yves Le Traon,et al.  Chapter Six - Mutation Testing Advances: An Analysis and Survey , 2019, Adv. Comput..

[8]  Jia-Guang Sun,et al.  EVMFuzz: Differential Fuzz Testing of Ethereum Virtual Machine , 2019, Journal of Software: Evolution and Process.

[9]  Marieke Huisman,et al.  Practical Mutation Testing for Smart Contracts , 2019, DPM/CBT@ESORICS.

[10]  Chao Peng,et al.  SIF: A Framework for Solidity Code Instrumentation and Analysis , 2019, ArXiv.

[11]  Andreas Zeller,et al.  (Un-)Covering Equivalent Mutants , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[12]  Haoran Wu,et al.  Mutation Testing for Ethereum Smart Contract , 2019, ArXiv.

[13]  Mark Harman,et al.  Detecting Trivial Mutant Equivalences via Compiler Optimisations , 2018, IEEE Transactions on Software Engineering.

[14]  Massimo Bartoletti,et al.  A Survey of Attacks on Ethereum Smart Contracts (SoK) , 2017, POST.

[15]  Tao Xie,et al.  Is operator-based mutant selection superior to random mutant selection? , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[16]  Andreas Zeller,et al.  The Impact of Equivalent Mutants , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[17]  K. N. King,et al.  A fortran language system for mutation‐based software testing , 1991, Softw. Pract. Exp..

[18]  Lei Ma,et al.  Oracle-Supported Dynamic Exploit Generation for Smart Contracts , 2019, IEEE Transactions on Dependable and Secure Computing.

[19]  Alex Groce,et al.  An Extensible, Regular-Expression-Based Tool for Multi-language Mutant Generation , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion).

[20]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[21]  Jonas Mellin,et al.  Test Case Generation for Mutation-based Testing of Timeliness , 2006, Electron. Notes Theor. Comput. Sci..

[22]  Russell O'Connor,et al.  Simplicity: A New Language for Blockchains , 2017, PLAS@CCS.

[23]  Suhabe Bugrara,et al.  User Experience with Language-Independent Formal Verification , 2019, ArXiv.

[24]  Dianxiang Xu,et al.  Deviant: A Mutation Testing Tool for Solidity Smart Contracts , 2019, 2019 IEEE International Conference on Blockchain (Blockchain).

[25]  Reyhaneh Jabbarvand,et al.  µDroid: an energy-aware mutation testing framework for Android , 2017, ESEC/SIGSOFT FSE.

[26]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[27]  Henry M. Kim,et al.  Understanding a Revolutionary and Flawed Grand Experiment in Blockchain: The DAO Attack , 2017, J. Cases Inf. Technol..

[28]  Alex Groce,et al.  Code coverage for suite evaluation by developers , 2014, ICSE.

[29]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[30]  Pieter Hartel,et al.  Truffle tests for free - Replaying Ethereum smart contracts for transparency , 2019, ArXiv.

[31]  A. Jefferson Offutt,et al.  Automatically detecting equivalent mutants and infeasible paths , 1997 .

[32]  Tibor Gyimóthy,et al.  Relating Code Coverage, Mutation Score and Test Suite Reducibility to Defect Density , 2016, 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[33]  Chao Peng,et al.  SIF: A Framework for Solidity Contract Instrumentation and Analysis , 2019, 2019 26th Asia-Pacific Software Engineering Conference (APSEC).