Model-based testing of networked applications

We present a principled automatic testing framework for application-layer protocols. The key innovation is a domain-specific embedded language for writing nondeterministic models of the behavior of networked servers. These models are defined within the Coq interactive theorem prover, supporting a smooth transition from testing to formal verification. Given a server model, we show how to automatically derive a tester that probes the server for unexpected behaviors. We address the uncertainties caused by both the server's internal choices and the network delaying messages nondeterministically. The derived tester accepts server implementations whose possible behaviors are a subset of those allowed by the nondeterministic model. We demonstrate the effectiveness of this framework by using it to specify and test a fragment of the HTTP/1.1 protocol, showing that the automatically derived tester can capture RFC violations in buggy server implementations, including the latest versions of Apache and Nginx.

[1]  Benjamin C. Pierce,et al.  Verifying an HTTP Key-Value Server with Interaction Trees and VST , 2021, ITP.

[2]  B. Pierce,et al.  Interaction trees: representing recursive and impure programs in Coq , 2019, Proc. ACM Program. Lang..

[3]  Tom Ridge,et al.  Engineering with Logic , 2018, J. ACM.

[4]  Benjamin C. Pierce,et al.  From C to interaction trees: specifying, verifying, and testing a networked server , 2018, CPP.

[5]  G. J. Tretmans,et al.  Model-Based Testing with TorXakis: The Mysteries of Dropbox Revisited , 2019 .

[6]  Wei Sun,et al.  Improving the cost-effectiveness of symbolic testing techniques for transport protocol implementations under packet dynamics , 2017, ISSTA.

[7]  Cyrille Artho,et al.  Model-based Testing of the Java Network API , 2017, FESCA@ETAPS.

[8]  Ulf Norell,et al.  Mysteries of DropBox: Property-Based Testing of a Distributed Synchronization Service , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[9]  Nobuko Yoshida,et al.  Protocol-based verification of message-passing parallel programs , 2015, OOPSLA.

[10]  Roy T. Fielding,et al.  Hypertext Transfer Protocol (HTTP/1.1): Conditional Requests , 2014, RFC.

[11]  Roy T. Fielding,et al.  Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content , 2014, RFC.

[12]  Cyrille Artho,et al.  Software model checking for distributed systems with selector-based, non-blocking communication , 2013, 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[13]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[14]  Tevfik Bultan,et al.  Netstub: a framework for verification of distributed java applications , 2007, ASE '07.

[15]  Mahesh Viswanathan,et al.  Finding Bugs in Network Protocols Using Simulation Code and Protocol-Specific Heuristics , 2005, ICFEM.

[16]  T. Kanade.  Model-Based Testing of Reactive Systems , 2005 .

[17]  Dawson R. Engler,et al.  Model Checking Large Network Protocol Implementations , 2004, NSDI.

[18]  Thierry Jéron,et al.  An Approach to Symbolic Test Generation , 2000, IFM.

[19]  Roy T. Fielding,et al.  The Apache HTTP Server Project , 1997, IEEE Internet Comput..

[20]  Matt Bishop,et al.  Property-based testing: a new approach to testing for assurance , 1997, SOEN.

[21]  Roy T. Fielding,et al.  Hypertext Transfer Protocol - HTTP/1.1 , 1997, RFC.

[22]  Tommaso Bolognesi,et al.  Tableau methods to describe strong bisimilarity on LOTOS processes involving pure interleaving and enabling , 1994, FORTE.