CoMan: Managing Bandwidth Across Computing Frameworks in Multiplexed Datacenters

Inefficient bandwidth sharing in a datacenter network, between different application frameworks, e.g., MapReduce and Spark, can lead to <italic>inelastic</italic> and <italic>skewed</italic> usage of link bandwidth and <italic> increased completion times</italic> for the applications. Existing work, however, either solely focuses on managing computation and storage resources or controlling only sending/receiving rate at hosts. In this paper, we present CoMan, a solution that provides global in-network bandwidth management in multiplexed data centers, with two goals: improving bandwidth utilization and reducing application completion time. CoMan first designs a novel abstraction of virtual link groups (VLGs) to establish a shared bandwidth resource pool. Based on this pool, CoMan implements a three-level bandwidth allocation model, which enables elastic bandwidth sharing among computing frameworks as well as guarantees network performance for the applications. CoMan further improves the bandwidth utilization by devising a VLG dependency graph and solves an optimization problem to guide the path selection using a <inline-formula> <tex-math notation="LaTeX">$\frac{3}{2}$</tex-math><alternatives><inline-graphic xlink:href="li-ieq1-2788003.gif"/> </alternatives></inline-formula>-approximation algorithm. We conduct comprehensive trace-driven simulations as well as small-scale testbed experiments to evaluate the performance of CoMan. Extensive simulation results show that CoMan improves the bandwidth utilization and speeds up the application completion time by up to <inline-formula> <tex-math notation="LaTeX">$2.83\mathcal{\times}$</tex-math><alternatives> <inline-graphic xlink:href="li-ieq2-2788003.gif"/></alternatives></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$6.68\mathcal{\times}$</tex-math><alternatives> <inline-graphic xlink:href="li-ieq3-2788003.gif"/></alternatives></inline-formula>, respectively, compared to the ECMP <inline-formula><tex-math notation="LaTeX">$+$</tex-math><alternatives> <inline-graphic xlink:href="li-ieq4-2788003.gif"/></alternatives></inline-formula> ElasticSwitch solution. Our implementation also verifies that CoMan can realistically speed up the application completion times by <inline-formula> <tex-math notation="LaTeX">$2.32\mathrm{\times}$</tex-math><alternatives> <inline-graphic xlink:href="li-ieq5-2788003.gif"/></alternatives></inline-formula> on average.