User Estimates Inaccuracy Study in HPC Scheduler

The article is focused on user estimates inaccuracy study in high performance computing scheduler. The main goal of this survey is to study historical job submission data from the real supercomputer cenre and investigate accuracy of resources estimated by user on job submission. Based on this study 68.94% of jobs used less than 25% of their requested walltime, which indicates a high estimation inaccuracy. In case of estimation of amount of memory only 31.46% of jobs uses more than 75% of requested memory. Analysis of historical jobs from the further prediction of resources accuracy has been made. Similarities of the new submitted jobs with historical data were described by keys based on user names, software name, walltime and number of processors. Also mathematical model for jobs scheduling in HPC system, with a goal functions, has been proposed and verified.

[1]  Andrea C. Arpaci-Dusseau,et al.  The Impact of More Accurate Requested Runtimes on Production Job Scheduling Performance , 2002, JSSPP.

[2]  Warren Smith,et al.  Using Run-Time Predictions to Estimate Queue Wait Times and Improve Scheduler Performance , 1999, JSSPP.

[3]  Zhiling Lan,et al.  Job scheduling with adjusted runtime estimates on production supercomputers , 2013, J. Parallel Distributed Comput..

[4]  Wojciech Bozejko,et al.  Parallel Genetic Algorithm for Minimizing Total Weighted Completion Time , 2004, ICAISC.

[5]  John E. West,et al.  Scheduling Jobs on Parallel Systems Using a Relaxed Backfill Strategy , 2002, JSSPP.

[6]  Francine Berman,et al.  A comprehensive model of the supercomputer workload , 2001 .

[7]  P. Sadayappan,et al.  Characterization of backfilling strategies for parallel job scheduling , 2002, Proceedings. International Conference on Parallel Processing Workshop.

[8]  Erik Elmroth,et al.  Towards understanding HPC users and systems: A NERSC case study , 2018, J. Parallel Distributed Comput..

[9]  Wojciech Boejko.  On single-walk parallelization of the job shop problem solving algorithms , 2012 .

[10]  Wojciech Bozejko,et al.  Block approach to the cyclic flow shop scheduling , 2015, Comput. Ind. Eng..

[11]  Dror G. Feitelson,et al.  Utilization, Predictability, Workloads, and User Runtime Estimates in Scheduling the IBM SP2 with Backfilling , 2001, IEEE Trans. Parallel Distributed Syst..

[12]  Wojciech Boejko,et al.  Solving the flow shop problem by parallel programming , 2009 .