SoftRDMA: Rekindling High Performance Software RDMA over Commodity Ethernet

Recent academic and industrial work is exploring the challenges of using RDMA over Ethernet, to support highly reliable, latency-sensitive services in today's datacenters. Previous work on the high-speed packet I/O like netmap, DPDK, etc., and high-performance user-level stacks like mTCP, IX etc., rekindles our inspirations to implement a high-performance software RDMA over commodity Ethernet devices. This paper summarizes and explores the much-overlapped design philosophy between RDMA and the high-performance user-level stacks. Inspired by these, we design SoftRDMA, which is a user-level iWARP stack, based on One-Copy and deliberate threading model design. SoftRDMA's system implementation includes user-level iWARP/TCP/IP protocols and the DPDK packet I/O. No special hardware or software is required beyond. It provides the basic verbs of iWARP for RDMA communication. In our evaluation, SoftRDMA demonstrates comparable latency and throughput performance against the hardware-supported iWARP scheme. It achieves microsecond latency for short message and nearly full line rate for long message transfer.

[1]  Brice Goglin.  Design and implementation of Open-MX: High-performance message passing over generic Ethernet hardware , 2008, 2008 IEEE International Symposium on Parallel and Distributed Processing.

[2]  Pavan Balaji,et al.  iWARP redefined: Scalable connectionless communication over high-speed Ethernet , 2010, 2010 International Conference on High Performance Computing.

[3]  Jinyang Li,et al.  Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store , 2013, USENIX ATC.

[4]  Christoforos E. Kozyrakis,et al.  IX: A Protected Dataplane Operating System for High Throughput and Low Latency , 2014, OSDI.

[5]  Eunyoung Jeong,et al.  mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems , 2014, NSDI.

[6]  Wenji Wu,et al.  The performance analysis of linux networking - Packet receiving , 2007, Comput. Commun..

[7]  Pete Wyckoff,et al.  iWarp protocol kernel space software implementation , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[8]  David Lee,et al.  Prevention of deadlocks and livelocks in lossless, backpressured packet networks , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[9]  Renato Recio,et al.  An RDMA Protocol Specification , 2002 .

[10]  Timothy Roscoe,et al.  Arrakis , 2014, OSDI.

[11]  Haitao Wu,et al.  RDMA over Commodity Ethernet at Scale , 2016, SIGCOMM.

[12]  Miguel Castro,et al.  FaRM: Fast Remote Memory , 2014, NSDI.

[13]  Luigi Rizzo,et al.  netmap: A Novel Framework for Fast Packet I/O , 2012, USENIX ATC.

[14]  Pete Wyckoff,et al.  Design and Implementation of the iWarp Protocol in Software , 2005, IASTED PDCS.

[15]  Ahmad Afsahi,et al.  10-Gigabit iWARP Ethernet: Comparative Performance Analysis with InfiniBand and Myrinet-10G , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[16]  Ankit Singla,et al.  Practical DCB for improved data center networks , 2014, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[17]  Mark Handley,et al.  Network stack specialization for performance , 2013, HotNets.

[18]  Sangjin Han,et al.  PacketShader: a GPU-accelerated software router , 2010, SIGCOMM '10.