Nimrod/K: Towards massively parallel dynamic Grid workflows

A challenge for Grid computing is the difficulty in developing software that is parallel, distributed and highly dynamic. Whilst there have been many general purpose mechanisms developed over the years, Grid programming still remains a low level, error prone task. Scientific workflow engines can double as programming environments, and allow a user to compose dasiavirtualpsila Grid applications from pre-existing components. Whilst existing workflow engines can specify arbitrary parallel programs, (where components use message passing) they are typically not effective with large and variable parallelism. Here we discuss dynamic dataflow, originally developed for parallel tagged dataflow architectures (TDAs), and show that these can be used for implementing Grid workflows. TDAs spawn parallel threads dynamically without additional programming. We have added TDAs to Kepler, and show that the system can orchestrate workflows that have large amounts of variable parallelism. We demonstrate the system using case studies in chemistry and in cardiac modelling.

[1]  David Abramson,et al.  Applying Grid Computing to the Parameter Sweep of a Group Difference Pseudopotential , 2004, International Conference on Computational Science.

[2]  Bertram Ludäscher,et al.  A Framework for the Design and Reuse of Grid Workflows , 2004, SAG.

[3]  David Abramson,et al.  Fractional factorial design for parameter sweep experiments using Nimrod/E , 2008, Sci. Program..

[4]  Ami Marowka,et al.  The GRID: Blueprint for a New Computing Infrastructure , 2000, Parallel Distributed Comput. Pract..

[5]  Rajkumar Buyya,et al.  A Taxonomy of Workflow Management Systems for Grid Computing , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[6]  Kaizar Amin,et al.  GridAnt: a client-controllable grid workflow system , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[7]  David Abramson,et al.  Application of grid computing to parameter sweeps and optimizations in molecular modeling , 2005, Future Gener. Comput. Syst..

[8]  Miron Livny,et al.  Condor: a distributed job scheduler , 2001 .

[9]  Edward A. Lee,et al.  Scientific workflow management and the Kepler system , 2006, Concurr. Comput. Pract. Exp..

[10]  Matthew R. Pocock,et al.  Taverna: a tool for the composition and enactment of bioinformatics workflows , 2004, Bioinform..

[11]  Mark S. Gordon,et al.  General atomic and molecular electronic structure system , 1993, J. Comput. Chem..

[12]  D. Abramson,et al.  An Automatic Design Optimization Tool and its Application to Computational Fluid Dynamics , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[13]  Arvind,et al.  Executing a Program on the MIT Tagged-Token Dataflow Architecture , 1990, IEEE Trans. Computers.

[14]  Ian T. Foster,et al.  Globus: a Metacomputing Infrastructure Toolkit , 1997, Int. J. High Perform. Comput. Appl..

[15]  Henri Casanova,et al.  Parameter Sweeps on the Grid with APST , 2003 .

[16]  Edward A. Lee,et al.  Composing Different Models of Computation in Kepler and Ptolemy II , 2007, International Conference on Computational Science.

[17]  Radu Prodan,et al.  ASKALON: a tool set for cluster and Grid computing , 2005, Concurr. Pract. Exp..

[18]  Edward A. Lee,et al.  Overview of the Ptolemy project , 2001 .

[19]  David E. Culler,et al.  Managing parallelism and resources in scientific dataflow programs , 1989 .

[20]  David Abramson,et al.  High performance parametric modeling with Nimrod/G: killer application for the global grid? , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[21]  M. Shields,et al.  Chapter 1 RESOURCE MANAGEMENT OF TRIANA P2P SERVICES , 2003 .

[22]  David Abramson,et al.  Parameter scan of an effective group difference pseudopotential using grid computing , 2009, New Generation Computing.

[23]  David Abramson,et al.  GridRod: a dynamic runtime scheduler for grid workflows , 2007, ICS '07.

[24]  Adam Arbree,et al.  Mapping Abstract Complex Workflows onto Grid Environments , 2003, Journal of Grid Computing.

[25]  David Abramson,et al.  The RMIT Data Flow Computer: A Hybrid Architecture , 1990, Comput. J..