A load balance oriented cost efficient scheduling method for parallel tasks

With the development of Internet technology, distributed task processing has become the key to solve the problems in big data computing, cloud computing, and collaborative computing. At the aspect of distributed task scheduling optimization, it is needed to establish the scheduling architecture with multiple schedulers, to meet the requirement of minimizing the cost of large scale parallel tasks. However the schedulers would give rise to the issue of high device load, intensive resource competition, and the inefficient collaboration. For this, we proposed the CESM (Cost Efficient Scheduling Method) method, which utilizes the weighted random schedule policy to assign the devices to the tasks, to reduce the competition of the task on the efficient low-cost devices. The weights in the random schedule process dependent on the scheduling environment, such as communication time, the busy state, the execution time and the cost. The efficient low-cost device tends to get a higher weight, implying it has a higher possibility to be assigned. That makes the scheduling results have a better rationality on execution time and cost. For this reason, we designed the weight model based on the communication time, the busy state, the execution time and the cost, and adopted the experimental method to analyze the values of the parameters. Finally, we gave four experiments on the arrival time test, device dependence test, task structure test, device set test, respectively, to verify the effectiveness and rationality of the proposed CESM.