Limago: An FPGA-Based Open-Source 100 GbE TCP/IP Stack

The realization that the network is becoming an important bottleneck in computing clusters and in the cloud has led in the past years to an increase scrutiny of how networking functionality is deployed. From TCP Offload Engines (TOEs) to Software Defined Networking (SDN), including Smart NICs and In-Network Data Processing, a wide range of approaches are currently being explored to increase the efficiency of networks and tailor its functionality to the actual needs of the application at hand. To address the need for an open and customizable networking stack, in this paper we introduce Limago, an FPGA-based open-source implementation of a TCP/IP stack operating at 100 Gbit/s. To our knowledge, Limago provides the first complete description of an FPGA-based TCP/IP stack at these speeds, thereby illustrating the bottlenecks that must be addressed, proposing several innovative designs to reach the necessary throughput, and showing how to incorporate advanced protocol features into the design. As an example, Limago supports the TCP Window Scale option, addressing the Long Fat Pipe issue. Limago not only enables 100 Gbit/s Ethernet links in an open source package, but also paves the way to programmable and fully customizable NICs based on FPGAs.

[1]  Enhong Chen,et al.  KV-Direct: High-Performance In-Memory Key-Value Store with Programmable NIC , 2017, SOSP.

[2]  Dhabaleswar K. Panda,et al.  Performance characterization of a 10-Gigabit Ethernet TOE , 2005, 13th Symposium on High Performance Interconnects (HOTI'05).

[3]  Andy Currid,et al.  TCP Offload to the Rescue , 2004, ACM Queue.

[4]  Li Ding,et al.  Hardware TCP Offload Engine based on 10-Gbps Ethernet for low-latency network communication , 2016, 2016 International Conference on Field-Programmable Technology (FPT).

[5]  Gustavo Alonso,et al.  Complex event detection at wire speed with FPGAs , 2010, Proc. VLDB Endow..

[6]  Hari Angepat,et al.  Serving DNNs in Real Time at Datacenter Scale with Project Brainwave , 2018, IEEE Micro.

[7]  Gustavo Alonso,et al.  Ibex - An Intelligent Storage Engine with Support for Advanced SQL Off-loading , 2014, Proc. VLDB Endow..

[8]  Van Jacobson,et al.  TCP Extensions for High Performance , 1992, RFC.

[9]  Kushagra Vaid,et al.  Azure Accelerated Networking: SmartNICs in the Public Cloud , 2018, NSDI.

[10]  Hari Angepat,et al.  A cloud-scale acceleration architecture , 2016, 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[11]  Gustavo Alonso,et al.  Low-latency TCP/IP stack for data center applications , 2016, 2016 26th International Conference on Field Programmable Logic and Applications (FPL).

[12]  G. Bioul,et al.  Synthesis of Arithmetic Circuits: FPGA, ASIC and Embedded Systems , 2006 .

[13]  Yong Ji,et al.  40Gbps multi-connection TCP/IP offload engine , 2011, 2011 International Conference on Wireless Communications and Signal Processing (WCSP).

[14]  Fernando Pedone,et al.  The Case For In-Network Computing On Demand , 2019, EuroSys.

[15]  Gustavo Alonso,et al.  FPGA-based TCP/IP Checksum Offloading Engine for 100 Gbps Networks , 2018, 2018 International Conference on ReConFigurable Computing and FPGAs (ReConFig).

[16]  Gustavo Alonso,et al.  Scalable 10Gbps TCP/IP Stack Architecture for Reconfigurable Hardware , 2015, 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines.

[17]  Gustavo Alonso,et al.  Caribou: Intelligent Distributed Storage , 2017, Proc. VLDB Endow..

[18]  Andrew Lord,et al.  The impact of capacity growth in national telecommunications networks , 2016, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[19]  Gustavo Alonso,et al.  Consensus in a Box: Inexpensive Coordination in Hardware , 2016, NSDI.

[20]  Klaus Hofmann,et al.  A 10 GbE TCP/IP hardware stack as part of a protocol acceleration platform , 2013, 2013 IEEE Third International Conference on Consumer Electronics ¿ Berlin (ICCE-Berlin).

[21]  Jon Postel,et al.  Transmission Control Protocol , 1981, RFC.

[22]  Salvatore Pontarelli,et al.  A Fully Portable TCP Implementation Using XFSMs , 2018, SIGCOMM Posters and Demos.

[23]  Andreas Herkersdorf,et al.  Enabling FPGAs in Hyperscale Data Centers , 2015, 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom).