Towards making formal methods normal: meeting developers where they are

Formal verification of software is a bit of a niche activity: it is only applied to the most safety-critical or security-critical software and it is typically only performed by specialized verification engineers. This paper considers whether it would be possible to increase adoption of formal methods by integrating formal methods with developers' existing practices and workflows. We do not believe that widespread adoption will follow from making the prevailing formal methods argument that correctness is more important than engineering teams realize. Instead, our focus is on what we would need to do to enable programmers to make effective use of formal verification tools and techniques. We do this by considering how we might make verification tooling that both serves developers' needs and fits into their existing development lifecycle. We propose a target of two orders of magnitude increase in adoption within a decade driven by ensuring a positive `weekly cost-benefit' ratio for developer time invested.

[1]  Phoebe V. Moore Metric power , 2018 .

[2]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[3]  Serdar Tasiran,et al.  Code-Level Model Checking in the Software Development Workflow , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[4]  Thomas R. G. Green,et al.  Cognitive dimensions of notations , 1990 .

[5]  A. Reid,et al.  End-to-End Verification of ARM ® Processors with ISA-Formal , 2016 .

[6]  Luke Church,et al.  API Usability at Scale , 2016, PPIG.

[7]  Emina Torlak,et al.  Finding code that explodes under symbolic evaluation , 2018, Proc. ACM Program. Lang..

[8]  K. Rustan M. Leino,et al.  Extended Static Checking: A Ten-Year Perspective , 2001, Informatics.

[9]  Josh Levenberg,et al.  Why Google stores billions of lines of code in a single repository , 2016, Commun. ACM.

[10]  Marita Franzke,et al.  Usability evaluation with the cognitive walkthrough , 1995, CHI 95 Conference Companion.

[11]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2000, ICFP.

[12]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[13]  Alan F. Blackwell,et al.  Cognitive Dimensions of Information Artefacts: a tutorial , 1998 .

[14]  Shriram Krishnamurthi,et al.  The Human in Formal Methods , 2019, FM.

[15]  Brian Cantwell Smith,et al.  On the origin of objects , 1997, Trends in Cognitive Sciences.

[16]  Xi Wang,et al.  An Empirical Study on the Correctness of Formally Verified Distributed Systems , 2017, EuroSys.

[17]  Patrice Godefroid,et al.  Fuzzing: hack, art, and science , 2020, Commun. ACM.

[18]  Alex Groce,et al.  DeepState: Symbolic Unit Testing for C and C++ , 2018 .

[19]  Diana K. Smetters,et al.  Moving from the design of usable security technologies to the design of useful secure applications , 2002, NSPW '02.

[20]  Luke Church,et al.  Modern Code Review: A Case Study at Google , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[21]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[22]  Zvonimir Rakamaric,et al.  Study of Integrating Random and Symbolic Testing for Object-Oriented Software , 2018, IFM.

[23]  Gogul Balakrishnan,et al.  Feedback-directed unit test generation for C/C++ using concolic execution , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[24]  Anna Philippou,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2018, Lecture Notes in Computer Science.

[25]  Peter Müller,et al.  Leveraging rust types for modular specification and verification , 2019, Proc. ACM Program. Lang..

[26]  K. Rustan M. Leino,et al.  Developing verified programs with Dafny , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[27]  Wei Wang,et al.  FUDGE: fuzz driver generation at scale , 2019, ESEC/SIGSOFT FSE.

[28]  Dirk Beyer,et al.  Software Verification: Testing vs. Model Checking - A Comparative Evaluation of the State of the Art , 2017, Haifa Verification Conference.

[29]  Willem Visser,et al.  COASTAL: Combining Concolic and Fuzzing for Java (Competition Contribution) , 2020, TACAS.

[30]  Ciera Jaspan,et al.  Tricorder: Building a Program Analysis Ecosystem , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[31]  Peter W. O'Hearn,et al.  Continuous Reasoning: Scaling the impact of formal methods , 2018, LICS.