Rekindling network protocol innovation with user-level stacks

Recent studies show that more than 86% of Internet paths allow well-designed TCP extensions, meaning that it is still possible to deploy transport layer improvements despite the existence of middleboxes in the network. Hence, the blame for the slow evolution of protocols (with extensions taking many years to nbecome widely used) should be placed on end systems. In this paper, we revisit the case for moving protocols stacks up into user space in order to ease the deployment of new protocols, extensions, or performance optimizations. We present MultiStack, operating system support for user-level protocol stacks. MultiStack runs within commodity operating systems, can concurrently host a large number of isolated stacks, has a fall-back path to the legacy host stack, and is able to process packets at rates of 10Gb/s. We validate our design by showing that our mux/demux layer can validate and switch packets at line rate (up to 14.88 Mpps) on a 10 Gbit port using 1-2 cores, and that a proof-of-concept HTTP server running over a basic userspace TCP outperforms by 18-90% both the same server and nginx running over the kernel's stack.

[1]  Donald E. Porter,et al.  Rethinking the library OS from the top down , 2011, ASPLOS XVI.

[2]  Chris Maeda,et al.  Networking performance for microkernels , 1992, [1992] Proceedings Third Workshop on Workstation Operating Systems.

[3]  Thu D. Nguyen,et al.  Implementing Network Protocols at User Level , 1993, SIGCOMM.

[4]  Mark Handley,et al.  How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP , 2012, NSDI.

[5]  Van Jacobson,et al.  TCP Extensions for High Performance , 1992, RFC.

[6]  Brian N. Bershad,et al.  Protocol service decomposition for high-performance networking , 1994, SOSP '93.

[7]  Peter Druschel,et al.  Lazy receiver processing (LRP): a network subsystem architecture for server systems , 1996, OSDI '96.

[8]  Byung-Gon Chun,et al.  Usenix Association 10th Usenix Symposium on Operating Systems Design and Implementation (osdi '12) 135 Megapipe: a New Programming Interface for Scalable Network I/o , 2022 .

[9]  Qian Zhang,et al.  A Compound TCP Approach for High-Speed and Long Distance Networks , 2006, Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications.

[10]  Antti Kantee,et al.  Environmental Independence : BSD Kernel TCP / IP in Userspace Antti Kantee , 2009 .

[11]  Thorsten von Eicken,et al.  U-Net: a user-level network interface for parallel and distributed computing , 1995, SOSP.

[12]  Mark Handley,et al.  The Case for Ubiquitous Transport-Level Encryption , 2010, USENIX Security Symposium.

[13]  Injong Rhee,et al.  CUBIC: a new TCP-friendly high-speed TCP variant , 2008, OPSR.

[14]  Sally Floyd,et al.  TCP Selective Acknowledgement Options , 1996 .

[15]  Sally Floyd,et al.  TCP Selective Acknowledgment Options , 1996, RFC.

[16]  Giuseppe Lettieri,et al.  VALE, a switched ethernet for virtual machines , 2012, CoNEXT '12.

[17]  Luigi Rizzo,et al.  netmap: A Novel Framework for Fast Packet I/O , 2012, USENIX ATC.

[18]  Luca Deri,et al.  High speed network traffic analysis with commodity multi-core systems , 2010, IMC '10.

[19]  Dawson R. Engler,et al.  Fast and flexible application-level networking on exokernel systems , 2002, TOCS.

[20]  Akira Kato,et al.  Traffic Data Repository at the WIDE Project , 2000, USENIX Annual Technical Conference, FREENIX Track.

[21]  Yuchung Cheng,et al.  TCP fast open , 2011, CoNEXT '11.

[22]  Mark Handley,et al.  Design, Implementation and Evaluation of Congestion Control for Multipath TCP , 2011, NSDI.

[23]  Alan Wagner,et al.  Portable and Performant Userspace SCTP Stack , 2012, 2012 21st International Conference on Computer Communications and Networks (ICCCN).

[24]  Mark Allman,et al.  Comments on bufferbloat , 2013, CCRV.

[25]  Mischa Schwartz,et al.  ACM SIGCOMM computer communication review , 2001, CCRV.

[26]  Mark Handley,et al.  Is it still possible to extend TCP? , 2011, IMC '11.