Scheduling Dependent Coflows with Guaranteed Job Completion Time

Today's data center jobs typically follow a coflow model. Each coflow consists of multiple concurrent data flows, while each job is comprised of multiple coflows. Only completing all flows in all coflows is meaningful to a job. To guarantee the job completion time, job deadlines and coflow dependencies must be jointly considered. However, existing solutions mainly consider the coflow scheduling, which are insufficient to guarantee the completion time of jobs with multiple dependent coflows. In this paper, we study the dependent coflow scheduling problem with constraints on job deadlines. Specifically, we formulate a deadline-and dependency-aware optimization problem, and accordingly propose a two-level scheduling method to solve this problem. The first level is to schedule at the job level with a most-bottleneck-first heuristic algorithm. The second level is an intra-job scheduling, which seamlessly combine a prioritized scheduling and a weighted fair scheduling, with the aim of accounting for different coflow dependencies. We conduct comprehensive simulations to evaluate the performance of our two-level scheduling method. Extensive results show that our scheduling method can reduce the job completion time by up to 18%, and accommodate 21% more jobs with deadlines guaranteed, compared to the conventional shortest-job-first method.