A cloud-scale per-flow backpressure system via FPGA-based heavy hitter detection

Virtual private clouds provide sharing resources to a massive number of tenants for economics of scale. In such clouds, off-the-shelf x86 boxes are widely deployed as network intermediate nodes. However, due to rapid growth of cloud traffic and significant slowdown of CPU improvement in recent years, although horizontal scaling is still leveraged, CPU overload and packet losses caused by heavy hitters are occasionally observed in production environment, which seriously damage tenant's SLAs. To address this, we propose a cloud-scale per-flow backpressure system designed in Alibaba Cloud. The basic idea is to (1) trigger the heavy-hitter flow acquisition at the intermediate node in an on-demand manner only when the CPU utilization exceeds a predefined threshold and (2) backpressure the identified heavy-hitter flow to the traffic source via rate limiting at sender's NIC or hypervisor. To handle the extremely large traffic rate of cloud traffic, we leverage a high-speed FPGA for heavy hitter detection acceleration. To accommodate highly concurrent flows in the cloud, we design a hierarchical memory system for accurate heavy hitter counting during a large time window. Under the per-flow backpressure mechanism, the rate of the heavy-hitter flow is accurately throttled while the rate of mice flows is completely unaffected during the backpressure.