The MicroGrid: using online simulation to predict application performance in diverse grid network environments

Improvements in networking and middleware technology are enabling large-scale grids that aggregate resources over wide-area networks to support applications at unprecedented levels of scale and performance. Unfortunately, existing middleware and tools provide little information to users as to the suitability of a given grid topology for a specific grid application. Instead, users generally use ad-hoc performance models to evaluate mappings of their applications to resource and network topologies. Grid application behavior alone is complex, and adding resource and network behavior makes the situation even worse. As a result, users typically employ nearly blind experimentation to find good deployments of their applications in each new grid environment. Only through actual deployment and execution can a user discovers if the mapping was a good one. Further, even after finding a good configuration, there is no basis to determine if a much better configuration has been missed. This approach slows effective grid application development and deployment. We present a richer methodology for evaluating grid software and diverse grid environments based on the MicroGrid grid online simulator. With the MicroGrid, users, grid researchers, or grid operators can define and simulate arbitrary collections of resources and networks. This allows study of an existing grid testbed under controlled conditions or even to study the efficacy of higher performance environments than are available today. Further, the MicroGrid supports direct execution of grid applications unchanged. These application can be written with MPI, C, C++, Perl, and/or Python and use the Globus middleware. This enables detailed and accurate study of application behavior. This work presents: (1) the first validation of the MicroGrid for studying whole-program performance of MPI grid applications and (2) a demonstration of the MicroGrid as a tool for predicting the performance of applications on a range of grid resources and novel network topologies.

[1]  Herb Schwetman,et al.  CSIM: a C-based process-oriented simulation language , 1986, WSC '86.

[2]  Andrew A. Chien,et al.  The MicroGrid: a Scientific Tool for Modeling Computational Grids , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[3]  Francine Berman,et al.  Grid Computing: Making the Global Infrastructure a Reality , 2003 .

[4]  James Cowie Hongbo Liu Jason Liu David Nicol Andy Ogielski.  Towards Realistic Million-Node Internet Simulations , 1999 .

[5]  Richard Barrett,et al.  Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods , 1994, Other Titles in Applied Mathematics.

[6]  Mario Gerla,et al.  GloMoSim: A Scalable Network Simulation Environment , 2002 .

[7]  Francine Berman,et al.  Overview of the Book: Grid Computing – Making the Global Infrastructure a Reality , 2003 .

[8]  Sathish S. Vadhiyar,et al.  Numerical Libraries and the Grid , 2001, Int. J. High Perform. Comput. Appl..

[9]  J. Cleary,et al.  SimKit: a high performance logical process simulation class library in C++ , 1995, Winter Simulation Conference Proceedings, 1995..

[10]  Rajkumar Buyya,et al.  GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for Grid computing , 2002, Concurr. Comput. Pract. Exp..

[11]  John A. Miller,et al.  JSIM: A Java-based simulation and animation environment , 1997, Proceedings of 1997 SCS Simulation Multiconference.

[12]  Luigi Rizzo,et al.  Dummynet: a simple approach to the evaluation of network protocols , 1997, CCRV.

[13]  D. Lipman,et al.  Improved tools for biological sequence comparison. , 1988, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Jason Maassen,et al.  Programming environments for high-performance Grid computing: the Albatross project , 2002, Future Gener. Comput. Syst..

[15]  Henri Casanova,et al.  Scheduling distributed applications: the SimGrid simulation framework , 2003, CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings..

[16]  Henri Casanova,et al.  A Simple MPI Process Swapping Architecture for Iterative Applications , 2004, Int. J. High Perform. Comput. Appl..

[17]  Deborah Estrin,et al.  Advances in network simulation , 2000, Computer.

[18]  B. Harrison.  Las Vegas, Nevada , 2002 .

[19]  Ian Foster,et al.  The Globus toolkit , 1998 .

[20]  Mike Hibler,et al.  An integrated experimental environment for distributed systems and networks , 2002, OSDI '02.

[22]  David M. Nicol,et al.  Towards Realistic Million-Node Internet Simulation , 1999, PDPTA.

[23]  Francine Berman,et al.  The GrADS Project: Software Support for High-Level Grid Application Development , 2001, Int. J. High Perform. Comput. Appl..

[24]  Gary William Flake,et al.  The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems and Adaptation , 1998 .

[25]  Miron Livny,et al.  Condor and the Grid , 2003 .

[26]  Henri Casanova,et al.  A decoupled scheduling approach for Grid application development environments , 2003, J. Parallel Distributed Comput..

[27]  Xin Liu,et al.  Traffic-based Load Balance for Scalable Network Emulation , 2003, ACM/IEEE SC 2003 Conference (SC'03).

[28]  Ami Marowka,et al.  The GRID: Blueprint for a New Computing Infrastructure , 2000, Parallel Distributed Comput. Pract..

[29]  Jack Dongarra,et al.  NetSolve: Past, Present, and Future - A Look at a Grid Enabled Server , 2003 .

[30]  Jack Dongarra,et al.  ScaLAPACK Users' Guide , 1987 .

[31]  David H. Bailey,et al.  The Nas Parallel Benchmarks , 1991, Int. J. High Perform. Comput. Appl..