Discovering Linear Models of Grid Workload

Despite extensive research focused on enabling QoS for grid users through economic and intelligent resource provisioning, no consensus has emerged on the most promising strategies. On top of intrinsically challenging problems, the complexity and size of data has so far drastically limited the number of comparative experiments. An alternative to experimenting on real, large, and complex data, is to look for well-founded and parsimonious representations. The goal of this paper is to answer a set of preliminary questions, which may help steering the design of those along feasible paths: is it possible to exhibit consistent models of the grid workload? If such models do exist, which classes of models are more appropriate, considering both simplicity and descriptive power? How can we actually discover such models? And finally, how can we assess the quality of these models on a statistically rigorous basis? Our main contributions are twofold. First we found that grid workload models can consistently be discovered from the real data, and that limiting the range of models to piecewise linear time series models is sufficiently powerful. Second, we presents a bootstrapping strategy for building more robust models from the limited samples at hand. This study is based on exhaustive information representative of a significant fraction of e-science computing activity in Europe.

[1]  Michèle Sebag,et al.  Toward autonomic grids: analyzing the job flow with affinity streaming , 2009, KDD.

[2]  Michael Muskulus,et al.  Analysis and modeling of job arrivals in a production grid , 2007, PERV.

[3]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[4]  Shantenu Jha,et al.  Investigating autonomic behaviours in grid-basedcomputational science applications , 2009, GMAC '09.

[5]  Richard Wolski,et al.  Eliciting honest value information in a batch-queue environment , 2007, 2007 8th IEEE/ACM International Conference on Grid Computing.

[6]  Jordi Torres,et al.  Maximizing revenue in Grid markets using an economically enhanced resource manager , 2010, Concurr. Comput. Pract. Exp..

[7]  Remzi H. Arpaci-Dusseau,et al.  Gathering at the Well: Creating Communities for Grid I/O , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[8]  Alexandru Iosup,et al.  Trace-based evaluation of job runtime and queue wait time predictions in grids , 2009, HPDC '09.

[9]  Radu Prodan,et al.  Soft Benchmarks-Based Application Performance Prediction Using a Minimum Training Set , 2006, 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06).

[10]  Jorma Rissanen,et al.  Stochastic Complexity in Statistical Inquiry , 1989, World Scientific Series in Computer Science.

[11]  Lorenza Saitta,et al.  Characterization of a computational grid as a complex system , 2009, GMAC '09.

[12]  Richard A. Davis,et al.  Structural Break Estimation for Nonstationary Time Series Models , 2006 .

[13]  Anand Raghunathan,et al.  Best-effort parallel execution framework for Recognition and mining applications , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[14]  Patrick Burns,et al.  Robustness of the Ljung-Box Test and its Rank Equivalent , 2002 .

[15]  Balázs Kégl,et al.  Utility-Based Reinforcement Learning for Reactive Grids , 2008, 2008 International Conference on Autonomic Computing.

[16]  David Vengerov,et al.  A Reinforcement Learning Approach to Dynamic Resource Allocation ∗ , 2005 .

[17]  Lingyun Yang,et al.  Conservative Scheduling: Using Predicted Variance to Improve Scheduling Decisions in Dynamic Environments , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[18]  Johan Montagnat,et al.  Modeling the latency on production grids with respect to the execution context , 2009, Parallel Comput..

[19]  Rajarshi Das,et al.  On the use of hybrid reinforcement learning for autonomic resource allocation , 2007, Cluster Computing.

[20]  Richard Wolski,et al.  Predicting Bounds on Queuing Delay in Space-shared Computing Environments , 2006, 2006 IEEE International Symposium on Workload Characterization.

[21]  Evgenia Smirni,et al.  Injecting realistic burstiness to a traditional client-server benchmark , 2009, ICAC '09.

[22]  Byoung-Dai Lee,et al.  Run-time prediction of parallel applications on shared environments , 2003, 2003 Proceedings IEEE International Conference on Cluster Computing.

[23]  R. Wolski,et al.  Predicting the CPU availability of time‐shared Unix systems on the computational grid , 1999, Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469).

[24]  Yang Yang,et al.  Bagging binary and quantile predictors for time series , 2006 .

[25]  Fabrizio Gagliardi,et al.  Building an infrastructure for scientific Grid computing: status and goals of the EGEE project , 2005, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[26]  Warren Smith,et al.  Using Run-Time Predictions to Estimate Queue Wait Times and Improve Scheduler Performance , 1999, JSSPP.

[27]  J. Gott.  Implications of the Copernican principle for our future prospects , 1993, Nature.

[28]  Allen B. Downey,et al.  Using Queue Time Predictions for Processor Allocation , 1997, JSSPP.

[29]  Peter A. Dinda,et al.  Host load prediction using linear models , 2000, Cluster Computing.