Job scheduling with adjusted runtime estimates on production supercomputers

Abstract The estimate of a parallel job’s running time (walltime) is an important attribute used by resource managers and job schedulers in various scenarios, such as backfilling and short-job-first scheduling. This value is provided by the user, however, and has been repeatedly shown to be inaccurate. We studied the workload characteristic based on a large amount of historical data (over 275,000 jobs in two and a half years) from a production leadership-class computer. Based on that study, we proposed a set of walltime adjustment schemes producing more accurate estimates. To ensure the utility of these schemes on production systems, we analyzed their potential impact in scheduling and evaluated the schemes with an event-driven simulator. Our experimental results show that our method can achieve not only better overall estimation accuracy but also improved overall system performance. Specifically, the average estimation accuracy of the tested workload can be improved by up to 35%, and the system performance in terms of average waiting time and weighted average waiting time can be improved by up to 22% and 28%, respectively.

[1]  Francine Berman,et al.  Using Moldability to Improve the Performance of Supercomputer Jobs , 2002, J. Parallel Distributed Comput..

[2]  Andrea C. Arpaci-Dusseau,et al.  The Impact of More Accurate Requested Runtimes on Production Job Scheduling Performance , 2002, JSSPP.

[3]  Guangwen Yang,et al.  PV-EASY: a strict fairness guaranteed and prediction enabled scheduler in parallel job scheduling , 2010, HPDC '10.

[4]  Warren Smith,et al.  Using Run-Time Predictions to Estimate Queue Wait Times and Improve Scheduler Performance , 1999, JSSPP.

[5]  Shonali Krishnaswamy,et al.  Estimating computation times of data-intensive applications , 2004, IEEE Distributed Systems Online.

[6]  Anand Sivasubramaniam,et al.  Improving parallel job scheduling by combining gang scheduling and backfilling techniques , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[7]  Zhiling Lan,et al.  Fault-aware, utility-based job scheduling on Blue, Gene/P systems , 2009, 2009 IEEE International Conference on Cluster Computing and Workshops.

[8]  Pavan Balaji,et al.  Improving Resource Availability by Relaxing Network Allocation Constraints on Blue Gene/P , 2009, 2009 International Conference on Parallel Processing.

[9]  Peter J. Keleher,et al.  Randomization, Speculation, and Adaptation in Batch Schedulers , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[10]  Warren Smith.  Prediction Services for Distributed Computing , 2007, 2007 IEEE International Parallel and Distributed Processing Symposium.

[11]  Hui Li,et al.  Predicting job start times on clusters , 2004, IEEE International Symposium on Cluster Computing and the Grid, 2004. CCGrid 2004..

[12]  Dan Tsafrir,et al.  Backfilling Using System-Generated Predictions Rather than User Runtime Estimates , 2007, IEEE Transactions on Parallel and Distributed Systems.

[13]  Zhiling Lan,et al.  Analyzing and adjusting user runtime estimates to improve job scheduling on the Blue Gene/P , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[14]  Carla E. Brodley,et al.  Predictive application-performance modeling in a computational grid environment , 1999, Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469).

[15]  Richard Gibbons,et al.  A Historical Application Profiler for Use by Parallel Schedulers , 1997, JSSPP.

[16]  John E. West,et al.  Scheduling Jobs on Parallel Systems Using a Relaxed Backfill Strategy , 2002, JSSPP.

[17]  Jens Volkert,et al.  An Architecture for an Adaptive Run-time Prediction System , 2008, 2008 International Symposium on Parallel and Distributed Computing.

[18]  Dan Tsafrir,et al.  The Dynamics of Backfilling: Solving the Mystery of Why Increased Inaccuracy May Help , 2006, 2006 IEEE International Symposium on Workload Characterization.

[19]  Cynthia Bailey Lee,et al.  Are User Runtime Estimates Inherently Inaccurate? , 2004, JSSPP.

[20]  Dror G. Feitelson,et al.  Utilization, Predictability, Workloads, and User Runtime Estimates in Scheduling the IBM SP2 with Backfilling , 2001, IEEE Trans. Parallel Distributed Syst..

[21]  Zhiling Lan,et al.  Reducing Fragmentation on Torus-Connected Supercomputers , 2011, 2011 IEEE International Parallel & Distributed Processing Symposium.

[22]  P. Sadayappan,et al.  Characterization of backfilling strategies for parallel job scheduling , 2002, Proceedings. International Conference on Parallel Processing Workshop.

[23]  Jens Volkert,et al.  A Three-Phase Adaptive Prediction System of the Run-Time of Jobs Based on User Behaviour , 2009, 2009 International Conference on Complex, Intelligent and Software Intensive Systems.

[24]  Dmitry N. Zotkin,et al.  Job-length estimation and performance in backfilling schedulers , 1999, Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469).

[25]  Dan Tsafrir,et al.  Modeling User Runtime Estimates , 2005, JSSPP.

[26]  Dan Tsafrir.  Using Inaccurate Estimates Accurately , 2010, JSSPP.

[27]  Francine Berman,et al.  A comprehensive model of the supercomputer workload , 2001 .

[28]  Kai Hwang,et al.  Adaptive Workload Prediction of Grid Performance in Confidence Windows , 2010, IEEE Transactions on Parallel and Distributed Systems.

[29]  Chita R. Das,et al.  A Fast and Efficient Processor Allocation Scheme for Mesh-Connected Multicomputers , 2002, IEEE Trans. Computers.

[30]  Allen B. Downey.  Predicting queue times on space-sharing parallel computers , 1997, Proceedings 11th International Parallel Processing Symposium.

[31]  Ian Foster,et al.  Predicting application run times with historical information , 2004, J. Parallel Distributed Comput..