Automated Test-Case Generation for Solidity Smart Contracts: the AGSolT Approach and its Evaluation

Blockchain and smart contract technology represent novel approaches to trusted services computing, opening the way to services designed specifically for trusted computing. Nevertheless, testing smart contracts is still in its infancy, with plenty of challenges not yet fully explored. We argue that existing tools are primarily for vulnerabilities detection and do not produce test suites suited for human oracles. In this paper, we present AGSOLT, a tool for Automated Generation of Solidity Test Suites. We evaluate the tool’s efficiency by implementing two search algorithms to automatically generate test suites for stand-alone Solidity smart contracts, considering some of the blockchain-specific challenges. Subsequently, to test AGSOLT in a realistic service operations scenario, we compared a random search algorithm and a genetic algorithm on a set of 36 real-world service applications featuring smart contracts. We found that AGSOLT is capable of achieving high branch coverage with both approaches and even discovered that some of the most popular Solidity smart contracts on GitHub have design flaws that might, for example, make code functions easily un-executable as a result of requiring too much gas. We conclude that AGSOLT provides a very valuable addition to service operations’ pipelines supporting trusted computing applications based on smart contracts.

[1]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[2]  Sina Shamshiri,et al.  Automated unit test generation for evolving software , 2015, ESEC/SIGSOFT FSE.

[3]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[4]  Gordon Fraser,et al.  A detailed investigation of the effectiveness of whole test suite generation , 2017, Empirical Software Engineering.

[5]  Ye Liu,et al.  ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[6]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[7]  Yang Feng,et al.  Smart Contract Development: Challenges and Opportunities , 2021, IEEE Transactions on Software Engineering.

[8]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[9]  Pedram Amini,et al.  Fuzzing: Brute Force Vulnerability Discovery , 2007 .

[10]  Haoran Wu,et al.  Mutation Testing for Ethereum Smart Contract , 2019, ArXiv.

[11]  Satoshi Nakamoto.  Bitcoin : A Peer-to-Peer Electronic Cash System , 2009 .

[12]  Paolo Tonella,et al.  A federated society of bots for smart contract testing , 2020, J. Syst. Softw..

[13]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[14]  Prateek Saxena,et al.  Making Smart Contracts Smarter , 2016, IACR Cryptol. ePrint Arch..

[15]  Tim Ebringer,et al.  ws-Attestation: Enabling Trusted Computing on Web Services , 2007, Test and Analysis of Web Services.

[16]  Harald C. Gall,et al.  An Empirical Investigation on the Readability of Manual and Generated Test Cases , 2018, 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC).

[17]  Luciano Baresi,et al.  TestFul: automatic unit-test generation for Java classes , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[18]  Haoran Wu,et al.  Towards Generating Cost-Effective Test-Suite for Ethereum Smart Contract , 2019, 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[19]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[20]  Gordon Fraser,et al.  Parameter tuning or default values? An empirical investigation in search-based software engineering , 2013, Empirical Software Engineering.

[21]  Pengcheng Zhang,et al.  SolidityCheck : Quickly Detecting Smart Contract Problems Through Regular Expressions , 2019, ArXiv.

[22]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[23]  Mary Lou Soffa,et al.  Generating test data for branch coverage , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[24]  Matthias Jarke,et al.  Data Sovereignty and Data Space Ecosystems , 2019, Business & Information Systems Engineering.

[25]  Ali Kashif Bashir,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2013, ICIRA 2013.

[26]  Gordon Fraser,et al.  Modeling readability to improve unit tests , 2015, ESEC/SIGSOFT FSE.

[27]  Robert E. Tarjan,et al.  A fast algorithm for finding dominators in a flowgraph , 1979, TOPL.

[28]  Yuanyuan Zhang,et al.  Achievements, Open Problems and Challenges for Search Based Software Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[29]  Lionel C. Briand,et al.  Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing , 2010, ICTSS.

[30]  Mario Köppen,et al.  Substitute Distance Assignments in NSGA-II for Handling Many-objective Optimization Problems , 2007, EMO.

[31]  Gordon Fraser,et al.  Random or evolutionary search for object‐oriented test suite generation? , 2018, Softw. Test. Verification Reliab..

[32]  Willem-Jan van den Heuvel,et al.  Model-Driven ML-Ops for Intelligent Enterprise Applications: Vision, Approaches and Challenges , 2020, BMSD.

[33]  Gordon Fraser,et al.  An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[34]  Carlos A. Brizuela,et al.  A survey on multi-objective evolutionary algorithms for many-objective problems , 2014, Computational Optimization and Applications.

[35]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[36]  Andrea De Lucia,et al.  Search-Based Testing of Procedural Programs: Iterative Single-Target or Multi-target Approach? , 2016, SSBSE.

[37]  Daniel Davis Wood,et al.  ETHEREUM: A SECURE DECENTRALISED GENERALISED TRANSACTION LEDGER , 2014 .

[38]  Damian A. Tamburri,et al.  Blockchains , 2019, ACM Comput. Surv..

[39]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[40]  Pengcheng Zhang,et al.  ADF-GA: Data Flow Criterion Based Test Case Generation for Ethereum Smart Contracts , 2020, ICSE.

[41]  Massimo Bartoletti,et al.  Financial Cryptography and Data Security , 2017, Lecture Notes in Computer Science.

[42]  Gordon Fraser,et al.  The Seed is Strong: Seeding Strategies in Search-Based Software Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[43]  Vitalik Buterin.  A NEXT GENERATION SMART CONTRACT & DECENTRALIZED APPLICATION PLATFORM , 2015 .

[44]  Massimo Bartoletti,et al.  A Survey of Attacks on Ethereum Smart Contracts (SoK) , 2017, POST.

[45]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[46]  Luca Guida,et al.  A Service-Oriented Perspective on Blockchain Smart Contracts , 2019, IEEE Internet Computing.

[47]  Chris J. Mitchell,et al.  Building General Purpose Security Services on Trusted Computing , 2011, INTRUST.

[48]  Paolo Tonella,et al.  Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets , 2018, IEEE Transactions on Software Engineering.

[49]  Joe D. Warren,et al.  The program dependence graph and its use in optimization , 1987, TOPL.

[50]  Marco Laumanns,et al.  Combining Convergence and Diversity in Evolutionary Multiobjective Optimization , 2002, Evolutionary Computation.

[51]  Yi Li,et al.  ModCon: a model-based testing platform for smart contracts , 2020, ESEC/SIGSOFT FSE.

[52]  Neel Sundaresan,et al.  Unit Test Case Generation with Transformers , 2020, ArXiv.

[53]  Mark Harman,et al.  A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search , 2010, IEEE Transactions on Software Engineering.

[54]  Ingo Weber,et al.  New kids on the block: an analysis of modern blockchains , 2016, ArXiv.

[55]  Paolo Tonella,et al.  LIPS vs MOSA: A Replicated Empirical Study on Automated Test Case Generation , 2017, SSBSE.