Resource use pattern analysis for predicting resource availability in opportunistic grids

This work presents a method for predicting resource availability in opportunistic grids by means of use pattern analysis (UPA), a technique based on non‐supervised learning methods. This prediction method is based on the assumption of the existence of several classes of computational resource use patterns, which can be used to predict the resource availability. Trace‐driven simulations validate this basic assumptions, which also provide the parameter settings for the accurate learning of resource use patterns. Experiments made with an implementation of the UPA method show the feasibility of its use in the scheduling of grid tasks with very little overhead. The experiments also demonstrate the method's superiority over other predictive and non‐predictive methods. An adaptative prediction method is suggested to deal with the lack of training data at initialization. Further adaptative behaviour is motivated by experiments which show that, in some special environments, reliable resource use patterns may not always be detected. Copyright © 2009 John Wiley & Sons, Ltd.

[1]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[2]  Guy E. Blelloch,et al.  Provably good multicore cache performance for divide-and-conquer algorithms , 2008, SODA '08.

[3]  Derek Wright,et al.  Cheap cycles from the desktop to the dedicated cluster: combining opportunistic and dedicated scheduling with Condor , 2007 .

[4]  Cynthia Bailey Lee,et al.  Are User Runtime Estimates Inherently Inaccurate? , 2004, JSSPP.

[5]  Matteo Frigo,et al.  The implementation of the Cilk-5 multithreaded language , 1998, PLDI.

[6]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[7]  Thierry Gautier,et al.  Deque-Free Work-Optimal Parallel STL Algorithms , 2008, Euro-Par.

[8]  Jarek Nabrzyski,et al.  Grid resource management: state of the art and future trends , 2004 .

[9]  Geoffrey E. Hinton,et al.  Unsupervised learning : foundations of neural computation , 1999 .

[10]  Graham R. Nudd,et al.  Pace—A Toolset for the Performance Prediction of Parallel and Distributed Systems , 2000, Int. J. High Perform. Comput. Appl..

[11]  Terrence J. Sejnowski,et al.  Unsupervised Learning , 2018, Encyclopedia of GIS.

[12]  Mary Shaw,et al.  Anticipatory configuration of resource-aware applications , 2005, EDSER '05.

[13]  David P. Anderson,et al.  On correlated availability in Internet-distributed systems , 2008, 2008 9th IEEE/ACM International Conference on Grid Computing.

[14]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[15]  P. Sopp Cluster analysis. , 1996, Veterinary immunology and immunopathology.

[16]  Miron Livny,et al.  Condor-a hunter of idle workstations , 1988, [1988] Proceedings. The 8th International Conference on Distributed.

[17]  Miron Livny,et al.  Scheduling Mixed Workloads in Multi-grids: The Grid Execution Hierarchy , 2006, 2006 15th IEEE International Conference on High Performance Distributed Computing.

[18]  Ian T. Foster,et al.  Globus: a Metacomputing Infrastructure Toolkit , 1997, Int. J. High Perform. Comput. Appl..

[19]  Thierry Gautier,et al.  KAAPI: A thread scheduling runtime system for data flow computations on cluster of multi-processors , 2007, PASCO '07.

[20]  Nguyen The Loc,et al.  Grid Scheduling using 2-Phase Prediction (2PP) of CPU Power , 2006, 2006 Innovations in Information Technology.

[21]  Fabio Kon,et al.  InteGrade: object‐oriented Grid middleware leveraging the idle computing power of desktop machines , 2004, Concurr. Pract. Exp..

[22]  Lingyun Yang,et al.  Conservative Scheduling: Using Predicted Variance to Improve Scheduling Decisions in Dynamic Environments , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[23]  Subhash Saini,et al.  Performance prediction and its use in parallel and distributed computing systems , 2006, Future Gener. Comput. Syst..

[24]  Ian T. Foster,et al.  Homeostatic and tendency-based CPU load predictions , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[25]  DeLiang Wang,et al.  Unsupervised Learning: Foundations of Neural Computation , 2001, AI Mag..

[26]  Nazareno Andrade,et al.  Labs of the World, Unite!!! , 2006, Journal of Grid Computing.

[27]  David P. Anderson,et al.  BOINC: a system for public-resource computing and storage , 2004, Fifth IEEE/ACM International Workshop on Grid Computing.

[28]  Miron Livny,et al.  Condor and preemptive resume scheduling , 2004 .