The MicroGrid : Using Emulation to Predict Application Performance in Diverse Grid Network Environments

Improvements in networking and middleware technology are enabling large-scale grids that aggregate resources over wide-area networks to support applications at unprecedented levels of scale and performance. Unfortunately, existing middleware and tools can not in general inform a user as to the suitability of a given grid topology for a specific grid application. Instead, users generally develop ad hoc performance models for mapping their applications to resource and network topologies. Because application behavior alone is complex, and resource and network behavior further complicate the picture, users are typically reduced to nearly blind experimentation to discover how to deploy their application in the new grid environment. Only after the fact can a user discovers if the match was a good one. Further, even after finding a desirable configuration, there is no basis on which to determine if a much better configuration exists. We present a richer methodology for evaluating grid software and diverse grid environments based on the MicroGrid grid emulator. With the MicroGrid, users, grid researchers, or grid operators can define and emulate arbitrary collections of resources and networks. This allows study of an existing grid testbed under controlled conditions or even to study the efficacy of higher performance environments than are available today. Further, the MicroGrid supports direct execution of grid applications unchanged. These application can be written with MPI, C, C++, Perl, and/or Python and use the Globus middleware. This enables detailed and accurate study of application behavior. 0This material is based upon work supported by the National Science Foundation under Grant #9975020 and #0103759. This paper presents: (1) the first validation of the MicroGrid for studying whole-program performance of MPI Grid applications and (2) a demonstration of the MicroGrid as a tool for predicting the performance of applications on a range of grid resources and novel network topologies.

[1]  Herb Schwetman,et al.  CSIM: a C-based process-oriented simulation language , 1986, WSC '86.

[2]  D. Lipman,et al.  Improved tools for biological sequence comparison. , 1988, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Richard Barrett,et al.  Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods , 1994, Other Titles in Applied Mathematics.

[4]  Fabian Gomes,et al.  SimKit: a high performance logical process simulation class library in C++ , 1995, WSC '95.

[5]  John A. Miller,et al.  JSIM: A Java-based simulation and animation environment , 1997, Proceedings of 1997 SCS Simulation Multiconference.

[6]  Luigi Rizzo,et al.  Dummynet: a simple approach to the evaluation of network protocols , 1997, CCRV.

[7]  Gary William Flake,et al.  The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems and Adaptation , 1998 .

[8]  Ian Foster,et al.  The Globus toolkit , 1998 .

[9]  James Cowie Hongbo Liu Jason Liu David Nicol Andy Ogielski.  Towards Realistic Million-Node Internet Simulations , 1999 .

[10]  Deborah Estrin,et al.  Advances in network simulation , 2000, Computer.

[11]  Andrew A. Chien,et al.  The MicroGrid: a Scientific Tool for Modeling Computational Grids , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[12]  G. Nigel Gilbert,et al.  The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems and Adaptation by Gary William Flake , 2000, J. Artif. Soc. Soc. Simul..

[13]  Ami Marowka,et al.  The GRID: Blueprint for a New Computing Infrastructure , 2000, Parallel Distributed Comput. Pract..

[14]  Sathish S. Vadhiyar,et al.  Numerical Libraries And The Grid: The GrADS Experiments With ScaLAPACK , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[15]  Francine Berman,et al.  The GrADS Project: Software Support for High-Level Grid Application Development , 2001, Int. J. High Perform. Comput. Appl..

[16]  Mike Hibler,et al.  An integrated experimental environment for distributed systems and networks , 2002, OPSR.

[17]  Jason Maassen,et al.  Programming environments for high-performance Grid computing: the Albatross project , 2002, Future Gener. Comput. Syst..

[18]  Rajkumar Buyya,et al.  GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing , 2002, Concurr. Comput. Pract. Exp..

[19]  Mario Gerla,et al.  GloMoSim: A Scalable Network Simulation Environment , 2002 .

[20]  Dejan Kostic,et al.  Scalability and accuracy in a large-scale network emulator , 2002, CCRV.

[21]  Jack Dongarra,et al.  NetSolve: Past, Present, and Future - A Look at a Grid Enabled Server , 2003 .

[22]  Francine Berman,et al.  Overview of the Book: Grid Computing – Making the Global Infrastructure a Reality , 2003 .

[23]  Xin Liu,et al.  Traffic-based Load Balance for Scalable Network Emulation , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[24]  Henri Casanova,et al.  A decoupled scheduling approach for Grid application development environments , 2003, J. Parallel Distributed Comput..

[25]  Henri Casanova,et al.  Scheduling distributed applications: the SimGrid simulation framework , 2003, CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings..

[26]  Miron Livny,et al.  Condor and the Grid , 2003 .

[27]  Henri Casanova,et al.  A Simple MPI Process Swapping Architecture for Iterative Applications , 2004, Int. J. High Perform. Comput. Appl..