Second Competition on Software Testing: Test-Comp 2020

This report describes the 2020 Competition on Software Testing (Test-Comp), the 2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^{\text {nd}}$$\end{document} edition of a series of comparative evaluations of fully automatic software test-case generators for C programs. The competition provides a snapshot of the current state of the art in the area, and has a strong focus on replicability of its results. The competition was based on 3 230 test tasks for C programs. Each test task consisted of a program and a test specification (error coverage, branch coverage). Test-Comp 2020 had 10 participating test-generation systems.

[1]  Joxan Jaffar,et al.  TracerX: Dynamic Symbolic Execution with Interpolation (Competition Contribution) , 2020, FASE.

[2]  Dirk Beyer,et al.  Rigorous examination of reactive systems , 2014, International Journal on Software Tools for Technology Transfer.

[3]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[4]  Raveendra Kumar Medicherla,et al.  VeriFuzz: Program Aware Fuzzing - (Competition Contribution) , 2019, TACAS.

[5]  Lucas C. Cordeiro,et al.  ESBMC: Scalable and Precise Test Generation based on the Floating-Point Theory , 2020, FASE.

[6]  Mark Harman,et al.  Testability transformation , 2004, IEEE Transactions on Software Engineering.

[7]  Urko Rueda,et al.  Java Unit Testing Tool Competition - Seventh Round , 2019, 2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST).

[8]  Lucas C. Cordeiro,et al.  Handling loops in bounded model checking of C programs via k-induction , 2015, International Journal on Software Tools for Technology Transfer.

[9]  Sarfraz Khurshid,et al.  Test input generation with java PathFinder , 2004, ISSTA '04.

[10]  Fabrice Kordon,et al.  TOOLympics 2019: An Overview of Competitions in Formal Methods , 2019, TACAS.

[11]  Dirk Beyer,et al.  Second Competition on Software Verification - (Summary of SV-COMP 2013) , 2013, TACAS.

[12]  Marie-Christine Jakobs.  CoVeriTest with Dynamic Partitioning of the Iteration Time Limit (Competition Contribution) , 2020, FASE.

[13]  Malte Lochau,et al.  HybridTiger: Hybrid Model Checking and Domination-based Partitioning for Efficient Multi-Goal Test-Suite Generation (Competition Contribution) , 2020, FASE.

[14]  Jorge A. Navas,et al.  TRACER: A Symbolic Execution Tool for Verification , 2012, CAV.

[15]  Cristian Cadar,et al.  KLEE symbolic execution engine in 2019 , 2020, International Journal on Software Tools for Technology Transfer.

[16]  Gidon Ernst,et al.  Legion: Best-First Concolic Testing (Competition Contribution) , 2020, FASE.

[17]  Dirk Beyer,et al.  Automatic Verification of C and Java Programs: SV-COMP 2019 , 2019, TACAS.

[18]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[19]  Koushik Sen,et al.  Combining Model Checking and Testing , 2018, Handbook of Model Checking.

[20]  Dirk Beyer,et al.  International Competition on Software Testing (Test-Comp) , 2019, TACAS.

[21]  Cesare Tinelli,et al.  Introducing StarExec: a Cross-Community Infrastructure for Logic Solving , 2014, COMPARE.

[22]  David Déharbe,et al.  The 2014 SMT Competition , 2014, J. Satisf. Boolean Model. Comput..

[23]  Helmut Veith,et al.  How did you specify your test suite , 2010, ASE.

[24]  Vladimir Klebanov,et al.  VerifyThis 2012 , 2015, International Journal on Software Tools for Technology Transfer.

[25]  Jan Strejcek,et al.  Joint Forces for Memory Safety Checking , 2018, SPIN.

[26]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[27]  Hoang M. Le,et al.  LLVM-based Hybrid Fuzzing with LibKluzzer (Competition Contribution) , 2020, FASE.

[28]  Dirk Beyer,et al.  CPU Energy Meter: A Tool for Energy-Aware Algorithms Engineering , 2020, TACAS.

[29]  Sven Apel,et al.  Facilitating Reuse in Multi-goal Test-Suite Generation for Software Product Lines , 2015, FASE.

[30]  Jean-Christophe Filliâtre,et al.  VerifyThis 2015 A program verification competition , 2016 .

[31]  Thomas Lemberger,et al.  Plain random test generation with PRTest , 2020, International Journal on Software Tools for Technology Transfer.

[32]  Dirk Beyer,et al.  Reliable benchmarking: requirements and solutions , 2017, International Journal on Software Tools for Technology Transfer.

[33]  Dirk Beyer,et al.  TestCov: Robust Test-Suite Execution and Coverage Measurement , 2019, 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[34]  Dirk Beyer,et al.  CoVeriTest: Cooperative Verifier-Based Testing , 2019, FASE.

[35]  Jim Alves-Foss,et al.  The DARPA Cyber Grand Challenge: A Competitor's Perspective, Part 2 , 2015, IEEE Security & Privacy.

[36]  Dirk Beyer,et al.  Software Verification: Testing vs. Model Checking - A Comparative Evaluation of the State of the Art , 2017, Haifa Verification Conference.

[37]  Geoff Sutcliffe.  The CADE ATP System Competition - CASC , 2016, AI Mag..