Empirical Study Towards a Leading Indicator for Cost of Formal Software Verification

Formal verification can provide the highest degree of software assurance. Demand for it is growing, but there are still few projects that have successfully applied it to sizeable, real-world systems. This lack of experience makes it hard to predict the size, effort and duration of verification projects. In this paper, we aim to better understand possible leading indicators of proof size. We present an empirical analysis of proofs from the landmark formal verification of the seL4 microkernel and the two largest software verification proof developments in the Archive of Formal Proofs. Together, these comprise 15,018 individual lemmas and approximately 215,000 lines of proof script. We find a consistent quadratic relationship between the size of the formal statement of a property, and the final size of its formal proof in the interactive theorem prover Isabelle. Combined with our prior work, which has indicated that there is a strong linear relationship between proof effort and proof size, these results pave the way for effort estimation models to support the management of large-scale formal verification projects.

[1]  E. James Whitehead,et al.  Managerial Issues for the Consideration and Use of Formal Methods , 2003, FME.

[2]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[3]  Gudmund Grov,et al.  Machine Learning in Proof General: Interfacing Interfaces , 2012, UITP.

[4]  Stefan Berghofer,et al.  Inductive Datatypes in HOL - Lessons Learned in Formal-Logic Engineering , 1999, TPHOLs.

[5]  Gudmund Grov.  A statistical relational learning challenge – extracting proof strategies from exemplar proofs , 2012 .

[6]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[7]  Kathleen Fisher.  HACMS: high assurance cyber military systems , 2012, HILT.

[8]  Gerwin Klein,et al.  seL4 Enforces Integrity , 2011, ITP.

[9]  David Delahaye,et al.  A Tactic Language for the System Coq , 2000, LPAR.

[10]  Markus Wenzel,et al.  An Isabelle Proof Method Language , 2014, ITP.

[11]  Andreas Lochbihler.  Jinja with Threads , 2007, Arch. Formal Proofs.

[12]  Steve King,et al.  Is Proof More Cost-Effective Than Testing? , 2000, IEEE Trans. Software Eng..

[13]  Tobias Nipkow,et al.  Proof Terms for Simply Typed Higher Order Logic , 2000, TPHOLs.

[14]  Lawrence Charles Paulson,et al.  Isabelle/HOL: A Proof Assistant for Higher-Order Logic , 2002 .

[15]  Timothy Bourke,et al.  seL4: From General Purpose to a Proof of Information Flow Enforcement , 2013, 2013 IEEE Symposium on Security and Privacy.

[16]  Martyn Thomas,et al.  Industrial deployment of system engineering methods providing high dependability and productivity. , 2013 .

[17]  David Aspinall,et al.  Proof General: A Generic Tool for Proof Development , 2000, TACAS.

[18]  Andreas Bollin,et al.  Metrics for quantifying evolutionary changes in Z specifications , 2013, J. Softw. Evol. Process..

[19]  Trustworthy Systems Team.  seL4 proofs for API 1.03, release 2014-08-10 , 2014 .

[20]  Leonard J. Bass,et al.  Formal specifications better than function points for code sizing , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[21]  Kaisa Sere,et al.  Specification Metrics for Event-B Developments , 2010 .

[22]  Filip Maric.  Formal Verification of Modern SAT Solvers , 2008, Arch. Formal Proofs.

[23]  Maurice H. Halstead,et al.  Elements of software science (Operating and programming systems series) , 1977 .

[24]  Kevin Fu,et al.  Pacemakers and Implantable Cardiac Defibrillators: Software Radio Attacks and Zero-Power Defenses , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[25]  Hovav Shacham,et al.  Comprehensive Experimental Analyses of Automotive Attack Surfaces , 2011, USENIX Security Symposium.

[26]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[27]  Gernot Heiser,et al.  Comprehensive formal verification of an OS microkernel , 2014, TOCS.

[28]  Toby C. Murray,et al.  Extensible Specifications for Automatic Re-use of Specifications and Proofs , 2012, SEFM.

[29]  Magnus O. Myreen,et al.  Translation validation for a verified OS kernel , 2013, PLDI.

[30]  W. B. Samson,et al.  Predictive software metrics based on a formal specification , 1987 .

[31]  Rupak Majumdar,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 1997, Lecture Notes in Computer Science.

[32]  D. Ross Jeffery,et al.  Productivity for proof engineering , 2014, ESEM '14.

[33]  D. Ross Jeffery,et al.  An empirical research agenda for understanding formal methods productivity , 2015, Inf. Softw. Technol..

[34]  Pierre Castéran,et al.  Interactive Theorem Proving and Program Development , 2004, Texts in Theoretical Computer Science An EATCS Series.

[35]  Abdollah Tabareh.  Predictive Software Measures Based on Formal Z Specifications , 2012 .

[36]  J. van Leeuwen,et al.  Theorem Proving in Higher Order Logics , 1999, Lecture Notes in Computer Science.

[37]  Yves Bertot,et al.  Interactive Theorem Proving and Program Development: Coq'Art The Calculus of Inductive Constructions , 2010 .