A distributed job scheduling and flow management system

Grid computing, as a specific model of distributed systems, has sparked recent interest in managing job execution among distributed resource domains. Introduction of the meta-scheduler is a key feature in grid evolution, and the next step is to achieve collaborative interactions between meta-schedulers within and external to organizational boundaries to achieve scalability, balanced resource utilization, and location transparency to job submitters. This paper details a distributed system design that consists of a collaborative meta-scheduling framework, and an expanded resource model with schedulers and data as resources. With this framework, we also explore job scheduling and data management issues, and investigate job flow and meta-scheduling interactions for new applications that require job execution beyond simple sequential and conditional control.

[1]  M. Shields,et al.  Chapter 1 RESOURCE MANAGEMENT OF TRIANA P2P SERVICES , 2003 .

[2]  Johan Montagnat,et al.  Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEUR , 2008, Int. J. High Perform. Comput. Appl..

[3]  Matthew R. Pocock,et al.  Taverna: a tool for the composition and enactment of bioinformatics workflows , 2004, Bioinform..

[4]  Koustuv Dasgupta,et al.  DECO: Data Replication and Execution CO-scheduling for Utility Grids , 2006, ICSOC.

[5]  Onyeka Ezenwoye,et al.  TRAP/BPEL - A Framework for Dynamic Adaptation of Composite Services , 2007, WEBIST.

[6]  Eric Gilbert,et al.  Virtual data Grid middleware services for data‐intensive science , 2006, Concurr. Comput. Pract. Exp..

[7]  Eduardo Huedo,et al.  A framework for adaptive execution in grids , 2004, Softw. Pract. Exp..

[8]  Frank Leymann,et al.  Choreography for the Grid: towards fitting BPEL to the resource framework , 2006, Concurr. Comput. Pract. Exp..

[9]  Liana L. Fong,et al.  BPEL4Job: A Fault-Handling Design for Job Flow Management , 2007, ICSOC.

[10]  Eric Gilbert,et al.  Virtual data Grid middleware services for data-intensive science: Research Articles , 2006 .

[11]  Francine Berman,et al.  The GrADS Project: Software Support for High-Level Grid Application Development , 2001, Int. J. High Perform. Comput. Appl..

[12]  Yanbin Liu,et al.  Looking for an Evolution of Grid Scheduling: Meta-Brokering , 2008 .

[13]  Matjaz B. Juric,et al.  Business process execution language for web services , 2004 .

[14]  Daniel S. Katz,et al.  Pegasus: A framework for mapping complex scientific workflows onto distributed systems , 2005, Sci. Program..

[15]  Jesús Labarta,et al.  eNANOS Grid Resource Broker , 2005, EGC.

[16]  Frank Leymann,et al.  Choreography for the Grid: towards fitting BPEL to the resource framework: Research Articles , 2006 .

[17]  Miron Livny,et al.  Condor and the Grid , 2003 .

[18]  Koustuv Dasgupta,et al.  Data-WISE: Efficient management of data-intensive workflows in scheduled grid environments , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.

[19]  Gargi Dasgupta,et al.  INFORM: integrated flow orchestration and meta-scheduling for managed grid systems , 2007, MC '07.

[20]  Jan Mendling,et al.  Business Process Execution Language for Web Services , 2006, EMISA Forum.

[21]  Liana L. Fong,et al.  Enabling Interoperability among Meta-Schedulers , 2008, 2008 Eighth IEEE International Symposium on Cluster Computing and the Grid (CCGRID).

[22]  Miron Livny,et al.  Stork: making data placement a first class citizen in the grid , 2004, 24th International Conference on Distributed Computing Systems, 2004. Proceedings..

[23]  Min Cai,et al.  A Peer-to-Peer Replica Location Service Based on a Distributed Hash Table , 2004, Proceedings of the ACM/IEEE SC2004 Conference.