A Fair Scheduling Algorithm for Adaptive Heterogeneous Resources in Data Centers

The resource scheduling problem of data center clusters has always been a hot topic in the field of cloud computing. Existing research efforts focus on fairness, resource utilization and energy efficiency, and lack of research on heterogeneous clustering issues. To solve the problem that the traditional DRF algorithm does not consider the classification of machine performance and task type, this paper proposes a fair scheduling algorithm X-DRF that adapts to heterogeneous resources in the data center. The algorithm mainly classifies the performance of physical machines, increases the machine performance scoring factor, and increases the training and job type judgment classification of the XGBoost model. The experiments show that CPU utilization and memory usage increased by 10% and 6%, respectively. The normalized ratio is increased by about 3% compared to the original DRF system. Therefore, the presented fair scheduling algorithm for heterogeneous resources is more fair and reasonable in terms of resource allocation.

[1]  Wei Wang,et al.  Multi-Resource Fair Allocation in Heterogeneous Cloud Computing Systems , 2015, IEEE Transactions on Parallel and Distributed Systems.

[2]  Andrew V. Goldberg,et al.  Quincy: fair scheduling for distributed computing clusters , 2009, SOSP '09.

[3]  Scott Shenker,et al.  Choosy: max-min fair sharing for datacenter jobs with constraints , 2013, EuroSys '13.

[4]  Christina Delimitrou,et al.  Quasar: resource-efficient and QoS-aware cluster management , 2014, ASPLOS.

[5]  Gu Jing,et al.  Predicting Misconfiguration-Induced Unsuccessful Executions of Jobs in Big Data System , 2017, 2017 IEEE 41st Annual Computer Software and Applications Conference (COMPSAC).

[6]  Jing Zhang,et al.  Cluster resource adjustment based on an improved artificial fish swarm algorithm in Mesos , 2016, 2016 IEEE 13th International Conference on Signal Processing (ICSP).

[7]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.

[8]  Haydar Ali Ismail,et al.  CPU and memory performance analysis on dynamic and dedicated resource allocation using XenServer in Data Center environment , 2016, 2016 2nd International Conference on Science and Technology-Computer (ICST).

[9]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[10]  Ioannis Lambadaris,et al.  Per-Server Dominant-Share Fairness (PS-DSF): A multi-resource fair allocation mechanism for heterogeneous servers , 2017, 2017 IEEE International Conference on Communications (ICC).

[11]  Xiangyu Li,et al.  Mystic: Predictive Scheduling for GPU Based Cloud Servers Using Machine Learning , 2016, 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS).

[12]  Arturo González-Escribano,et al.  uBench: exposing the impact of CUDA block geometry in terms of performance , 2013, The Journal of Supercomputing.

[13]  Benjamin Hindman,et al.  Dominant Resource Fairness: Fair Allocation of Multiple Resource Types , 2011, NSDI.

[14]  Baochun Li,et al.  Dominant resource fairness in cloud computing systems with heterogeneous servers , 2013, IEEE INFOCOM 2014 - IEEE Conference on Computer Communications.

[15]  Senén Barro,et al.  Do we need hundreds of classifiers to solve real world classification problems? , 2014, J. Mach. Learn. Res..

[16]  Patrick Wendell,et al.  Sparrow: distributed, low latency scheduling , 2013, SOSP.