An Adaptive Partition Method for Handling Skew in Spark Applications

In the parallel computing framework of Hadoop/Spark, data skew is a common problem resulting in performance degradation, such as prolonging of the entire execution time and idle resources. What lies behind this issue is partition imbalance, which causes significant differences in the amount of data processed by each reduce task. This paper proposes a key reassigning and splitting partition algorithm (SKRSP) to handle skew, which considers both the partition balance of the intermediate data and the partition balance after shuffle operator. We design two partition algorithms for different applications: the range-based key splitting partition method (KSRP) for sort operation and hash-based key reassigning partition method (KRHP) for the other operations. We implement SKRSP in Spark 2.2.0 and evaluate its performance through three benchmarks exhibiting significant data skew: Sort, Join, and PageRank. The experimental results verify that our algorithm not only can achieve a better partition balance but also reduce the execution time of reduce tasks effectively.