Application-specific scheduling for the organic grid

Summary form only given. We propose a biologically inspired and fully-decentralized approach to the organization of computation that is based on the autonomous scheduling of strongly mobile agents on a peer-to-peer network. Our approach achieves the following design objectives: near-zero knowledge of network topology, zero knowledge of system status, autonomous scheduling, distributed computation, lack of specialized nodes. Every node is equally responsible for scheduling and computation, both of which are performed with practically no information about the system. We believe that this model is ideally suited for large-scale unstructured grids such as desktop grids. This model avoids the extensive system knowledge requirements of traditional grid scheduling approaches. Contrary to the popular master/worker organization of current desktop grids, our approach does not rely on specialized super-servers or on application-specific clients. By encapsulating computation and scheduling behavior into mobile agents, we decouple both application code and scheduling functionality from the underlying infrastructure. The resulting system is one where every node can start a large grid job, and where the computation naturally organizes itself around available resources. Through the careful design of agent behavior, the resulting global organization of the computation can be customized for different classes of applications. In a previous paper, we described a proof-of-concept prototype for an independent task application. We generalize the scheduling framework and demonstrate that our approach is applicable to a computation with a highly synchronous communication pattern, namely Cannon's matrix multiplication.

[1]  John F. Shoch,et al.  The “worm” programs—early experience with a distributed computation , 1982, CACM.

[2]  E. Bonabeau,et al.  Spatial patterns in ant colonies , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Stephen J. Wright,et al.  Near-optimal adaptive control of a large grid application , 2002, ICS '02.

[4]  Francine Berman,et al.  Adaptive Computing on the Grid Using AppLeS , 2003, IEEE Trans. Parallel Distributed Syst..

[5]  Ian T. Foster,et al.  MPICH-G2: A Grid-enabled implementation of the Message Passing Interface , 2002, J. Parallel Distributed Comput..

[6]  Jarek Nabrzyski,et al.  Grid Resource Management , 2004 .

[7]  Marco Dorigo,et al.  Swarm intelligence: from natural to artificial systems , 1999 .

[8]  David Abramson,et al.  High performance parametric modeling with Nimrod/G: killer application for the global grid? , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[9]  Richard Wolski,et al.  Analyzing Market-Based Resource Allocation Strategies for the Computational Grid , 2001, Int. J. High Perform. Comput. Appl..

[10]  Miron Livny,et al.  Adaptive Scheduling for Master-Worker Applications on the Computational Grid , 2000, GRID.

[11]  Ian T. Foster,et al.  Condor-G: A Computation Management Agent for Multi-Institutional Grids , 2004, Cluster Computing.

[12]  Lynn Elliot Cannon,et al.  A cellular computer to implement the kalman filter algorithm , 1969 .

[13]  R. F. Freund,et al.  Dynamic matching and scheduling of a class of independent tasks onto heterogeneous computing systems , 1999, Proceedings. Eighth Heterogeneous Computing Workshop (HCW'99).

[14]  A. M. Turing,et al.  The chemical basis of morphogenesis , 1952, Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences.

[15]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM 2001.

[16]  Andrew S. Grimshaw,et al.  The Legion vision of a worldwide virtual computer , 1997, Commun. ACM.

[17]  Mario Lauria,et al.  The organic grid: self-organizing computation on a peer-to-peer network , 2004 .

[18]  Larry Carter,et al.  Autonomous protocols for bandwidth-centric scheduling of independent-task applications , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[19]  Miron Livny,et al.  Condor-a hunter of idle workstations , 1988, [1988] Proceedings. The 8th International Conference on Distributed.

[20]  Peter M. A. Sloot,et al.  Hiërarchical Job Scheduling for Clusters of Workstations , 2000 .

[21]  P. D. Coddington,et al.  Scheduling Independent Tasks on Metacomputing Systems , 1999 .

[22]  H. Meinhardt,et al.  A theory of biological pattern formation , 1972, Kybernetik.

[23]  Andrew A. Chien,et al.  Entropia: architecture and performance of an enterprise desktop grid system , 2003, J. Parallel Distributed Comput..

[24]  Hein Meling,et al.  Messor: Load-Balancing through a Swarm of Autonomous Agents , 2002, AP2PC.

[25]  Robert Morris,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM 2001.

[26]  Tim Kindberg,et al.  Adaptive parallelism under Equus , 1994, Proceedings of 2nd International Workshop on Configurable Distributed Systems.