Prediction-Based and Locality-Aware Task Scheduling for Parallelizing Video Transcoding Over Heterogeneous MapReduce Cluster

MapReduce is a popular programming model in cloud computing to deal with the high computational task, such as video transcoding. It splits the video (task) into multiple segments (subtasks) and transcodes them in parallel in cluster. Due to the complexity of video transcoding and the poor performance of heterogeneous MapReduce cluster, scheduling these subtasks to minimize the total transcoding time is still a challenge. In this paper, we propose a prediction-based and locality-aware task scheduling (PLTS) method for parallelizing video transcoding over heterogeneous MapReduce cluster. First, we analyze video decoding and encoding technologies and predict the segment transcoding complexity, which can provide a foundational base for the following scheduling. Second, we attempt to schedule subtasks on machines that contain the related input data, which are referred to as data locality, so as to reduce large-scale data movement and data transfer during the mapping phase. Third, we formulate the scheduling as a job shop scheduling problem and propose a heuristic PLTS algorithm. It combines the benefits of two traditional heuristic scheduling algorithms, Max–Min and Min–Min, to make load balancing in cluster and short the total transcoding time. The experimental results also show the efficiency of our algorithm.