PIXIDA: Optimizing Data Parallel Jobs in Wide-Area Data Analytics

In the era of global-scale services, big data analytical queries are often required to process datasets that span multiple data centers (DCs). In this setting, cross-DC bandwidth is often the scarcest, most volatile, and/or most expensive resource. However, current widely deployed big data analytics frameworks make no attempt to minimize the traffic traversing these links. In this paper, we present Pixida, a scheduler that aims to minimize data movement across resource constrained links. To achieve this, we introduce a new abstraction called Silo, which is key to modeling Pixida's scheduling goals as a graph partitioning problem. Furthermore, we show that existing graph partitioning problem formulations do not map to how big data jobs work, causing their solutions to miss opportunities for avoiding data movement. To address this, we formulate a new graph partitioning problem and propose a novel algorithm to solve it. We integrated Pixida in Spark and our experiments show that, when compared to existing schedulers, Pixida achieves a significant traffic reduction of up to ~ 9x on the aforementioned links.