TOOLympics 2019: An Overview of Competitions in Formal Methods

Evaluation of scientific contributions can be done in many different ways. For the various research communities working on the verification of systems (software, hardware, or the underlying involved mechanisms), it is important to bring together the community and to compare the state of the art, in order to identify progress of and new challenges in the research area. Competitions are a suitable way to do that.

[1]  Fabrice Kordon,et al.  BenchKit, a Tool for Massive Concurrent Benchmarking , 2014, 2014 14th International Conference on Application of Concurrency to System Design.

[2]  Dirk Beyer,et al.  Reuse of Verification Results - Conditional Model Checking, Precision Reuse, and Verification Witnesses , 2013, SPIN.

[3]  Geoff Sutcliffe,et al.  The CADE-20 Automated Theorem Proving Competition , 2006, AI Commun..

[4]  Dirk Beyer,et al.  Status Report on Software Verification - (Competition Summary SV-COMP 2014) , 2014, TACAS.

[5]  Geoff Sutcliffe.  The CADE-23 Automated Theorem Proving System Competition - CASC-23 , 2012, AI Commun..

[6]  Fabrice Kordon,et al.  Raw Report on the Model Checking Contest at Petri Nets 2012 , 2012, ArXiv.

[7]  Dirk Beyer,et al.  BDD-based software verification , 2014, International Journal on Software Tools for Technology Transfer.

[8]  Florian Lonsing,et al.  The QBF Gallery: Behind the scenes , 2016, Artif. Intell..

[9]  Hubert Garavel,et al.  Benchmarking Implementations of Term Rewriting and Pattern Matching in Algebraic, Functional, and Object-Oriented Languages - The 4th Rewrite Engines Competition , 2018, WRLA@ETAPS.

[10]  Geoff Sutcliffe.  The IJCAR-2004 Automated Theorem Proving Competition , 2005, AI Commun..

[11]  Dirk Beyer,et al.  Software Verification: Testing vs. Model Checking - A Comparative Evaluation of the State of the Art , 2017, Haifa Verification Conference.

[12]  Jaco van de Pol,et al.  Sound Black-Box Checking in the LearnLib , 2018, NFM.

[13]  Grigore Rosu,et al.  Rewriting Logic Systems , 2007, WRLA.

[14]  Fabrice Kordon,et al.  MCC'2017 - The Seventh Model Checking Contest , 2018, Trans. Petri Nets Other Model. Concurr..

[15]  Johannes Waldmann.  Report on the Termination Competition 2008 , 2009 .

[16]  Geoff Sutcliffe.  The 5th IJCAR automated theorem proving system competition - CASC-J5 , 2011, AI Commun..

[17]  Fabrice Kordon,et al.  Presentation of the 9th Edition of the Model Checking Contest , 2019, TACAS.

[18]  Makoto Hamana,et al.  Confluence Competition 2018 , 2018, FSCD.

[19]  Ezio Bartocci,et al.  First International Competition on Software for Runtime Verification , 2014, RV.

[20]  Dirk Beyer,et al.  Rigorous examination of reactive systems , 2014, International Journal on Software Tools for Technology Transfer.

[21]  Theo C. Ruys,et al.  Thoughtful brute-force attack of the RERS 2012 and 2013 Challenges , 2014, International Journal on Software Tools for Technology Transfer.

[22]  Francisco Durán,et al.  The Third Rewrite Engines Competition , 2010, WRLA.

[23]  Geoff Sutcliffe,et al.  The development of CASC , 2002, AI Commun..

[24]  Cristina Serban,et al.  SL-COMP: Competition of Solvers for Separation Logic , 2019, TACAS.

[25]  E. Poll,et al.  A Program Verification Competition , 2015 .

[26]  Dirk Beyer,et al.  Competition on Software Verification - (SV-COMP) , 2012, TACAS.

[27]  Edward A. Hirsch,et al.  The SAT2002 competition , 2004, Annals of Mathematics and Artificial Intelligence.

[28]  Mihaela Sighireanu,et al.  Report on SL-COMP 2014 , 2014, J. Satisf. Boolean Model. Comput..

[29]  Dirk Beyer,et al.  Reliable and Reproducible Competition Results with BenchExec and Witnesses (Report on SV-COMP 2016) , 2016, TACAS.

[30]  Albert Oliveras,et al.  Design and Results of the 3rd Annual Satisfiability Modulo Theories Competition (SMT-Comp 2007) , 2008, Int. J. Artif. Intell. Tools.

[31]  Francisco Durán,et al.  The Rewrite Engines Competitions: A RECtrospective , 2019, TACAS.

[32]  Roberto Bruttomesso,et al.  The 2012 SMT Competition , 2013, SMT@IJCAR.

[33]  Nao Hirokawa,et al.  Confluence Competition 2015 , 2015, CADE.

[34]  Dirk Beyer,et al.  Software Verification with Validation of Results - (Report on SV-COMP 2017) , 2017, TACAS.

[35]  Fabrice Kordon,et al.  Report on the Model Checking Contest at Petri Nets 2011 , 2012, Trans. Petri Nets Other Model. Concurr..

[36]  Alfons Laarman,et al.  LTSmin: High-Performance Language-Independent Model Checking , 2015, TACAS.

[37]  Laurent Simon,et al.  Fifty-Five Solvers in Vancouver: The SAT 2004 Competition , 2004, SAT (Selected Papers.

[38]  Dirk Beyer,et al.  Second Competition on Software Verification - (Summary of SV-COMP 2013) , 2013, TACAS.

[39]  Dirk Beyer,et al.  Witness validation and stepwise testification across software verifiers , 2015, Software Engineering.

[40]  Sylvain Hallé,et al.  Third International Competition on Runtime Verification - CRV 2016 , 2016, RV.

[41]  Aaron Stump,et al.  Design and Results of the First Satisfiability Modulo Theories Competition (SMT-COMP 2005) , 2005, Journal of Automated Reasoning.

[42]  Robert Nieuwenhuis.  The impact of CASC in the development of automated deduction systems , 2002, AI Commun..

[43]  Dirk Beyer,et al.  Reliable benchmarking: requirements and solutions , 2017, International Journal on Software Tools for Technology Transfer.

[44]  Jean-Christophe Filliâtre,et al.  VerifyThis 2015 A program verification competition , 2016 .

[45]  Lucas C. Cordeiro,et al.  Applying symbolic bounded model checking to the 2012 RERS greybox challenge , 2014, International Journal on Software Tools for Technology Transfer.

[46]  Albert Rubio,et al.  Termination Competition (termCOMP 2015) , 2015, CADE.

[47]  Geoff Sutcliffe.  The CADE-21 automated theorem proving system competition , 2008, AI Commun..

[48]  Vladimir Klebanov,et al.  Evaluating Software Verification Systems: Benchmarks and Competitions (Dagstuhl Reports 14171) , 2014, Dagstuhl Reports.

[49]  Dirk Beyer,et al.  The RERS Grey-Box Challenge 2012: Analysis of Event-Condition-Action Systems , 2012, ISoLA.

[50]  Josef Urban,et al.  The CADE-25 Automated Theorem Proving system competition - CASC-25 , 2016, AI Commun..

[51]  Vladimir Klebanov,et al.  VerifyThis 2012 , 2015, International Journal on Software Tools for Technology Transfer.

[52]  Bernhard Steffen,et al.  Property-Preserving Parallel Decomposition , 2017, Models, Algorithms, Logics and Tools.

[53]  Tiziana Margaria,et al.  Property-driven benchmark generation: synthesizing programs of realistic structure , 2014, International Journal on Software Tools for Technology Transfer.

[54]  Carsten Sinz,et al.  Overview and analysis of the SAT Challenge 2012 solver competition , 2015, Artif. Intell..

[55]  Geoff Sutcliffe.  The 8th IJCAR automated theorem proving system competition - CASC-J8 , 2016, AI Commun..

[56]  Albert Rubio,et al.  The Termination and Complexity Competition , 2019, TACAS.

[57]  Bernhard Steffen,et al.  Rigorous Examination of Reactive Systems: The RERS Challenge 2015 , 2015, RV.

[58]  Aaron Stump,et al.  SMT-COMP: Satisfiability Modulo Theories Competition , 2005, CAV.

[59]  Francisco Durán,et al.  The Second Rewrite Engines Competition , 2009, WRLA.

[60]  Geoff Sutcliffe.  The CADE ATP System Competition - CASC , 2016, AI Mag..

[61]  Sofie Haesaert,et al.  ARCH-COMP18 Category Report: Stochastic Modelling , 2018, ARCH@ADHS.

[62]  Laurent Simon,et al.  The Essentials of the SAT 2003 Competition , 2003, SAT.

[63]  Geoff Sutcliffe,et al.  The state of CASC , 2006, AI Commun..

[64]  Marijn J. H. Heule,et al.  SAT Competition 2016: Recent Developments , 2017, AAAI.

[65]  Geoff Sutcliffe.  The 4th IJCAR Automated Theorem Proving System Competition - CASC-J4 , 2009, AI Commun..

[66]  Geoff Sutcliffe,et al.  The Results - of the CADE-13 ATP System Competition , 2004, Journal of Automated Reasoning.

[67]  Gordon J. Pace,et al.  An International Workshop on Competitions , Usability , Benchmarks , Evaluation , and Standardisation for Runtime Verification Tools Runtime Verification using Larva , 2017 .

[68]  Geoff Sutcliffe,et al.  The CADE-14 ATP System Competition , 1998, Journal of Automated Reasoning.

[69]  Julian Nagele,et al.  Confluence Competition 2019 , 2019, TACAS.

[70]  Bernhard Steffen,et al.  RERS 2018: CTL, LTL, and Reachability , 2018, ISoLA.

[71]  Bernhard Steffen,et al.  From ZULU to RERS - Lessons Learned in the ZULU Challenge , 2010, ISoLA.

[72]  Dirk Beyer.  Software Verification and Verifiable Witnesses - (Report on SV-COMP 2015) , 2015, TACAS.

[73]  Geoff Sutcliffe,et al.  The CADE-19 ATP System Competition , 2004, AI Commun..

[74]  Aaron Stump,et al.  The 2013 Evaluation of SMT-COMP and SMT-LIB , 2015, Journal of Automated Reasoning.

[75]  Tiziana Margaria,et al.  Property-Driven Benchmark Generation , 2013, SPIN.

[76]  Leonid Ryzhyk,et al.  The first reactive synthesis competition (SYNTCOMP 2014) , 2015, International Journal on Software Tools for Technology Transfer.

[77]  Geoff Sutcliffe.  The CADE-22 automated theorem proving system competition - CASC-22 , 2010, AI Commun..

[78]  Bernhard Steffen,et al.  Property-Preserving Generation of Tailored Benchmark Petri Nets , 2017, 2017 17th International Conference on Application of Concurrency to System Design (ACSD).

[79]  Albert Oliveras,et al.  6 Years of SMT-COMP , 2012, Journal of Automated Reasoning.

[80]  Geoff Sutcliffe,et al.  The IJCAR ATP System Competition , 2004, Journal of Automated Reasoning.

[81]  Geoff Sutcliffe.  The CADE-24 automated theorem proving system competition - CASC-24 , 2014, AI Commun..

[82]  Vladimir Klebanov,et al.  VerifyThis Verification Competition 2012 - Organizer's Report , 2013 .

[83]  Adrian Prantl,et al.  Combining static analysis and state transition graphs for verification of event-condition-action systems in the RERS 2012 and 2013 challenges , 2014, International Journal on Software Tools for Technology Transfer.

[84]  Geoff Sutcliffe.  The CADE-26 automated theorem proving system competition - CASC-26 , 2017, AI Commun..

[85]  Frits W. Vaandrager,et al.  RERS 2019: Combining Synthesis with Real-World Models , 2019, TACAS.

[86]  Jan Kretínský,et al.  The 2019 Comparison of Tools for the Analysis of Quantitative Formal Models - (QComp 2019 Competition Report) , 2019, TACAS.

[87]  Dirk Beyer,et al.  Automatic Verification of C and Java Programs: SV-COMP 2019 , 2019, TACAS.

[88]  Aaron Stump,et al.  Design and results of the 2nd annual satisfiability modulo theories competition (SMT-COMP 2006) , 2007, Formal Methods Syst. Des..

[89]  Geoff Sutcliffe,et al.  The CADE-13 ATP System Competition , 2004, Journal of Automated Reasoning.

[90]  Bernhard Steffen,et al.  RERS 2016: Parallel and Sequential Benchmarks with Focus on LTL Verification , 2016, ISoLA.

[91]  Cesare Tinelli,et al.  Introducing StarExec: a Cross-Community Infrastructure for Logic Solving , 2014, COMPARE.

[92]  Vladimir Klebanov,et al.  VerifyThis 2015 , 2016, International Journal on Software Tools for Technology Transfer.

[93]  Bernhard Steffen,et al.  The RERS 2017 challenge and workshop (invited paper) , 2017, SPIN.

[94]  Ezio Bartocci,et al.  International Competition on Runtime Verification (CRV) , 2019, TACAS.

[95]  Ezio Bartocci,et al.  First international Competition on Runtime Verification: rules, benchmarks, tools, and final results of CRV 2014 , 2017, International Journal on Software Tools for Technology Transfer.

[96]  Olivier Roussel,et al.  The International SAT Solver Competitions , 2012, AI Mag..

[97]  Geoff Sutcliffe.  The 7th IJCAR automated theorem proving system competition - CASC-J7 , 2015, AI Commun..

[98]  Erik Poll,et al.  VerifyThis 2016: A Program Verification Competition , 2016 .

[99]  Tiziana Margaria,et al.  Tailored generation of concurrent benchmarks , 2014, International Journal on Software Tools for Technology Transfer.

[100]  Geoff Sutcliffe.  The 9th IJCAR Automated Theorem Proving System Competition - CASC-J9 , 2018, AI Commun..

[101]  Geoff Sutcliffe,et al.  The CADE-15 ATP System Competition , 1999, Journal of Automated Reasoning.

[102]  Geoff Sutcliffe.  The 3rd IJCAR Automated Theorem Proving Competition , 2007, AI Commun..

[103]  Geoff Sutcliffe.  The 6th IJCAR automated theorem proving system competition - CASC-J6 , 2013, AI Commun..

[104]  Dirk Beyer,et al.  Correctness witnesses: exchanging verification results between verifiers , 2016, SIGSOFT FSE.

[105]  Gidon Ernst,et al.  VerifyThis - Verification Competition with a Human Factor , 2019, TACAS.

[106]  Dejan Nickovic,et al.  Second International Competition on Runtime Verification CRV 2015 , 2016, RV.

[107]  Hans Zantema,et al.  The Termination Competition , 2007, RTA.

[108]  Fabrice Kordon,et al.  Model Checking Contest @ Petri Nets, Report on the 2013 edition , 2013, ArXiv.

[109]  Dirk Beyer,et al.  International Competition on Software Testing (Test-Comp) , 2019, TACAS.

[110]  Gidon Ernst,et al.  The COST IC0701 Verification Competition 2011 , 2011, FoVeOOS.

[111]  David Déharbe,et al.  The 2014 SMT Competition , 2014, J. Satisf. Boolean Model. Comput..