Hypertesting : The Case for Automated Testing of Hyperproperties

Proof systems give absolute guarantees but are notoriously difficult to use for non-experts. Bug-finding tools make no completeness guarantees but offer a high degree of automation and are relatively easy to use for developers. For safety properties, the effectiveness of automatic test generation and bug finding is well established. For security properties like non-interference, which cannot be expressed as properties of a single program execution (i.e., hyperproperties), methods for testing and bug finding are in their infancy. In general, violations of hyperproperties cannot be expressed with a single test case like safety properties, so existing bug finding methods do not apply. This paper takes the position that we should fill this gap in the arsenal of verification technology and outlines concepts and tools for the next generation of bug finding systems. In particular, we aim to establish a generalized concept for the generation of “hypertests”, sets of tests that either provide some level of confidence in the system or give counterexamples to hyperproperties. As concrete instances of hypertesting, we foresee automated testing for violations of secure information flow and of numeric and cryptographic properties of programs.

[1]  Gorjan Alagic,et al.  #p , 2019, Quantum information & computation.

[2]  Michael Backes,et al.  Computationally sound verification of source code , 2010, CCS '10.

[3]  Steve Hanna,et al.  A Symbolic Execution Framework for JavaScript , 2010, 2010 IEEE Symposium on Security and Privacy.

[4]  Alfredo Pironti,et al.  Implementing TLS with Verified Cryptographic Security , 2013, 2013 IEEE Symposium on Security and Privacy.

[5]  A. W. Roscoe.  CSP and determinism in security modelling , 1995, Proceedings 1995 IEEE Symposium on Security and Privacy.

[6]  Arnar Birgisson,et al.  Boosting the Permissiveness of Dynamic Information-Flow Tracking by Testing , 2012, ESORICS.

[7]  Bernd Finkbeiner,et al.  Relational abstract interpretation for the verification of 2-hypersafety properties , 2013, CCS.

[8]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[9]  Andrew D. Gordon,et al.  Refinement Types for Secure Implementations , 2008, 2008 21st IEEE Computer Security Foundations Symposium.

[10]  Alexander Aiken,et al.  Secure Information Flow as a Safety Problem , 2005, SAS.

[11]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[12]  Corina S. Pasareanu,et al.  Reliability analysis in Symbolic PathFinder , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[13]  Danny Dolev,et al.  On the security of public key protocols , 1981, 22nd Annual Symposium on Foundations of Computer Science (sfcs 1981).

[14]  J. Meseguer,et al.  Security Policies and Security Models , 1982, 1982 IEEE Symposium on Security and Privacy.

[15]  Pedro R. D'Argenio,et al.  Secure information flow by self-composition , 2004, Proceedings. 17th IEEE Computer Security Foundations Workshop, 2004..

[16]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[17]  Marcelo d'Amorim,et al.  Quantifying information leaks using reliability analysis , 2014, SPIN.

[18]  Alejandro Russo,et al.  From Dynamic to Static and Back: Riding the Roller Coaster of Information-Flow Control Research , 2009, Ershov Memorial Conference.

[19]  Benjamin Grégoire,et al.  Probabilistic relational verification for cryptographic implementations , 2014, POPL.

[20]  Michael Norrish,et al.  seL4: formal verification of an OS kernel , 2009, SOSP '09.

[21]  Michael R. Clarkson,et al.  Hyperproperties , 2008, 2008 21st IEEE Computer Security Foundations Symposium.

[22]  Benjamin C. Pierce,et al.  Testing noninterference, quickly , 2016, Journal of Functional Programming.