Improving the cost-effectiveness of symbolic testing techniques for transport protocol implementations under packet dynamics

The majority of Internet traffic is transferred by transport protocols. The correctness of these transport protocol implementations is hard to validate as their behaviors depend not only on their protocols but also on their network environments that can introduce dynamic packet delay and loss. Random testing, widely used in industry due to its simplicity and low cost, struggles to detect packet delay related faults which occur with low probability. Symbolic execution based testing is promising at detecting such low probability faults, but it requires large testing budgets as it attempts to cover a prohibitively large input space of packet dynamics. To improve its cost-effectiveness, we propose two domain-specific heuristic techniques, called packet retransmission based priority and network state based priority, which are motivated by two common transport protocol properties. In our experiments using the Linux TFTP programs, our techniques improve the cost-effectiveness of symbolic execution based testing for transport protocols, detecting three times as many faults when the budget is in the range of minutes and hours.

[1]  Ramesh Govindan,et al.  Finding protocol manipulation attacks , 2011, SIGCOMM.

[2]  Matthew B. Dwyer,et al.  Safely reducing the cost of unit level symbolic execution through read/write analysis , 2014, SOEN.

[3]  Stephen McCamant,et al.  Statically-directed dynamic automated test generation , 2011, ISSTA '11.

[4]  Koushik Sen,et al.  MultiSE: multi-path symbolic execution using value summaries , 2015, ESEC/SIGSOFT FSE.

[5]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[6]  Ana R. Cavalli,et al.  Using passive testing based on symbolic execution and slicing techniques: Application to the validation of communication protocols , 2013, Comput. Networks.

[7]  Ana R. Cavalli,et al.  FSM-based conformance testing methods: A survey annotated with experimental evaluation , 2010, Inf. Softw. Technol..

[8]  Qian Zhang,et al.  A Compound TCP Approach for High-Speed and Long Distance Networks , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[9]  Katerina J. Argyraki,et al.  Software dataplane verification , 2014, NSDI.

[10]  Injong Rhee,et al.  CUBIC: a new TCP-friendly high-speed TCP variant , 2008, OPSR.

[11]  Robert T. Braden,et al.  Requirements for Internet Hosts - Application and Support , 1989, RFC.

[12]  Gary Scott Malkin,et al.  TFTP Option Extension , 1998, RFC.

[13]  Tianlong Yu,et al.  BUZZ: Testing Context-Dependent Policies in Stateful Networks , 2016, NSDI.

[14]  Jeffrey S. Foster,et al.  Rule-based static analysis of network protocol implementations , 2006, Inf. Comput..

[15]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[16]  George Candea,et al.  Efficient state merging in symbolic execution , 2012, Software Engineering.

[17]  Lisong Xu,et al.  SPD: Automatically Test Unmodified Network Programs with Symbolic Packet Dynamics , 2014, GLOBECOM 2014.

[18]  Zhendong Su,et al.  Steering symbolic execution to less traveled paths , 2013, OOPSLA.

[19]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[20]  Michael Hicks,et al.  Directed Symbolic Execution , 2011, SAS.

[21]  Costin Raiciu,et al.  SymNet: static checking for stateful networks , 2013, HotMiddlebox '13.

[22]  Dawson R. Engler,et al.  Redundant State Detection for Dynamic Symbolic Execution , 2013, USENIX Annual Technical Conference.

[23]  Stéphane Maag,et al.  A Formal Data-Centric Approach for Passive Testing of Communication Protocols , 2013, IEEE/ACM Transactions on Networking.

[24]  Dawson R. Engler,et al.  Model Checking Large Network Protocol Implementations , 2004, NSDI.

[25]  Michael Norrish,et al.  Engineering with logic: HOL specification and symbolic-evaluation testing for TCP implementations , 2006, POPL '06.

[26]  Costin Raiciu,et al.  SymNet: Scalable symbolic execution for modern networks , 2016, SIGCOMM.

[27]  Brad Richards,et al.  A framework for model checking UDP network programs with Java pathfinder , 2014, HILT.

[28]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[29]  Peter Schachte,et al.  State Joining and Splitting for the Symbolic Execution of Binaries , 2009, RV.

[30]  Marco Canini,et al.  A NICE Way to Test OpenFlow Applications , 2012, NSDI.

[31]  Richard Lai,et al.  A survey of communication protocol testing , 2002, J. Syst. Softw..

[32]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[33]  Peter R. Pietzuch,et al.  SymbexNet: Testing Network Protocol Implementations with Symbolic Execution and Rule-Based Specifications , 2014, IEEE Transactions on Software Engineering.

[34]  Alexandre Petrenko,et al.  Protocol testing: review of methods and relevance for software testing , 1994, ISSTA '94.

[35]  Klaus Wehrle,et al.  KleeNet: discovering insidious interaction bugs in wireless sensor networks before deployment , 2010, IPSN '10.

[36]  David Brumley,et al.  Enhancing symbolic execution with veritesting , 2014, ICSE.

[37]  David Lee,et al.  Network protocol system monitoring-a formal approach with passive testing , 2006, IEEE/ACM Transactions on Networking.

[38]  Klaus Wehrle,et al.  Integration testing of protocol implementations using symbolic distributed execution , 2012, 2012 20th IEEE International Conference on Network Protocols (ICNP).

[39]  Ramesh Govindan,et al.  Analyzing Protocol Implementations for Interoperability , 2015, NSDI.

[40]  Dawson R. Engler,et al.  RWset: Attacking Path Explosion in Constraint-Based Test Generation , 2008, TACAS.

[41]  George Candea,et al.  The S2E Platform: Design, Implementation, and Applications , 2012, TOCS.

[42]  Wei Sun,et al.  SPD: Automatically Test Unmodified Network Programs with Symbolic Packet Dynamics , 2014, 2015 IEEE Global Communications Conference (GLOBECOM).

[43]  Gary Scott Malkin,et al.  TFTP Blocksize Option , 1998, RFC.

[44]  Andreas Terzis,et al.  packetdrill: Scriptable Network Stack Testing, from Sockets to Packets , 2013, USENIX Annual Technical Conference.

[45]  Patrice Godefroid,et al.  Compositional dynamic test generation , 2007, POPL '07.

[46]  Osman Hasan,et al.  Applying Formal Methods to Networking: Theory, Techniques, and Applications , 2013, IEEE Communications Surveys & Tutorials.

[47]  Sunghun Kim,et al.  How we get there: a context-guided search strategy in concolic testing , 2014, SIGSOFT FSE.

[48]  Amarnath Mukherjee,et al.  On the Dynamics and Significance of Low Frequency Components of Internet Load , 1992 .

[49]  Matthew B. Dwyer,et al.  Automatic generation of load tests , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[50]  Svetlana Prokopenko,et al.  Detecting faults in TFTP implementations using Finite State Machines with timeouts , 2012 .

[51]  Klaus Wehrle,et al.  Scalable Symbolic Execution of Distributed Systems , 2011, 2011 31st International Conference on Distributed Computing Systems.

[52]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[53]  Vern Paxson,et al.  End-to-end Internet packet dynamics , 1997, SIGCOMM '97.

[54]  Koushik Sen,et al.  Symbolic execution for software testing: three decades later , 2013, CACM.

[55]  Soojin Park,et al.  Enhancing Conformance Testing Using Symbolic Execution for Network Protocols , 2015, IEEE Transactions on Reliability.

[56]  Gary Scott Malkin,et al.  TFTP Timeout Interval and Transfer Size Options , 1998, RFC.