A Study of Big Data Computing Platforms: Fairness and Energy Consumption

Improving the performance is the common sense on those large-scale data processing frameworks and fruitful studies are proposed in this direction. In contrast, the fairness and energy consumption of those frameworks need further exploration and how the performance, fairness and energy consumption interact each other on big data computing frameworks is not well addressed. In our research, we study the fairness and the energy consumption of those big data computing systems. We find that there are tradeoff between these factors. We conduct detailed studies on the factors which impact the tradeoff between different factors. Based on the observations in our study, we propose workload aware, energy-efficient and green-aware optimizations and implement them into Hadoop YARN. Particularly, in this thesis proposal, we propose to explore the following research problems. First, we explore the tradeoff between fairness and performance, and improve the performance of the state-of the-art approach by up to 225% [7]. Second, we consider the energy efficiency, renewable energy supply as well as battery usage and reduce the brown energy consumption of existing systems by more than 25% [8]. Third, we will explore the relationship between fairness and energy consumption, and eventually we will develop multi-objective optimizations for performance, fairness and energy consumption.