Grid benchmarking: vision, challenges, and current status

Grid benchmarking is an important and challenging topic of Grid computing research. In this paper, we present an overview of the key challenges that need to be addressed for the integration of benchmarking practices, techniques, and tools in emerging Grid computing infrastructures. We discuss the problems of performance representation, measurement, and interpretation in the context of Grid benchmarking, and propose the use of ontologies for organizing and describing benchmarking metrics. Finally, we present a survey of ongoing research efforts that develop benchmarks and benchmarking tools for the Grid. Copyright © 2006 John Wiley & Sons, Ltd.

[1]  N. F. Noy,et al.  Ontology Development 101: A Guide to Creating Your First Ontology , 2001 .

[2]  Marios D. Dikaiakos,et al.  GridBench: a tool for benchmarking grids , 2003, Proceedings. First Latin American Web Congress.

[3]  David H. Bailey,et al.  The Nas Parallel Benchmarks , 1991, Int. J. High Perform. Comput. Appl..

[4]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[5]  Anthony J. G. Hey,et al.  PARKBENCH: Methodology, Relations and Results , 1996, HPCN Europe.

[6]  Marian Bubak,et al.  An interactive grid environment for non-invasive vascular reconstruction , 2004 .

[7]  Philip Koopman,et al.  Dependability Benchmarking: making choices in an n-dimensional problem space , 2001 .

[8]  Stephen A. Jarvis,et al.  Performance‐based middleware for Grid computing , 2005, Concurr. Pract. Exp..

[9]  Peter M. A. Sloot,et al.  Grid Resource Selection by Application Benchmarking: a Computational Haemodynamics Case Study , 2005 .

[10]  Michael A. Frumkin,et al.  NAS Grid Benchmarks: a tool for Grid space exploration , 2001, Proceedings 10th IEEE International Symposium on High Performance Distributed Computing.

[11]  Richard Wolski,et al.  The network weather service: a distributed resource performance forecasting service for metacomputing , 1999, Future Gener. Comput. Syst..

[12]  Roderic G. G. Cattell.  The benchmark handbook for database and transaction processing systems , 1991 .

[13]  Jack J. Dongarra,et al.  Performance of various computers using standard linear equations software in a FORTRAN environment , 1988, CARN.

[14]  Carl Kesselman,et al.  Concepts and Architecture , 2004, The Grid 2, 2nd Edition.

[15]  Margo I. Seltzer,et al.  A self-scaling and self-configuring benchmark for Web servers (extended abstract) , 1998, SIGMETRICS '98/PERFORMANCE '98.

[16]  Marios D. Dikaiakos,et al.  GridBench: A Workbench for Grid Benchmarking , 2005, EGC.

[17]  Ying Liu,et al.  Understanding Grid resource information management through a synthetic database benchmark/workload , 2004, IEEE International Symposium on Cluster Computing and the Grid, 2004. CCGrid 2004..

[18]  Wayne Liu,et al.  Chapter 14 – Service Virtualization: Infrastructure and Applications , 2004 .

[19]  Ian T. Foster,et al.  The Anatomy of the Grid: Enabling Scalable Virtual Organizations , 2001, Int. J. High Perform. Comput. Appl..

[20]  Anoop Gupta,et al.  The SPLASH-2 programs: characterization and methodological considerations , 1995, ISCA.

[21]  Michael A. Frumkin,et al.  Evaluating the Information Power Grid using the NAS Grid Benchmarks , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[22]  Jim Gray,et al.  Benchmark Handbook: For Database and Transaction Processing Systems , 1992 .

[23]  Marios D. Dikaiakos,et al.  Performance Evaluation of Mobile-Agent Middleware: A Hierarchical Approach , 2001, Mobile Agents.

[24]  J. Gustafson.  Purpose-Based Benchmarks , 2004, Int. J. High Perform. Comput. Appl..

[25]  Francine Berman,et al.  The GrADS Project: Software Support for High-Level Grid Application Development , 2001, Int. J. High Perform. Comput. Appl..

[26]  Marios D. Dikaiakos,et al.  Mobile agent platforms for Web databases: a qualitative and quantitative assessment , 1999, Proceedings. First and Third International Symposium on Agent Systems Applications, and Mobile Agents.

[27]  Simon See,et al.  Benchmark performance on cluster grid with NGB , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[28]  Daniel P. Siewiorek,et al.  Comparing operating systems using robustness benchmarks , 1997, Proceedings of SRDS'97: 16th IEEE Symposium on Reliable Distributed Systems.

[29]  David A. Patterson,et al.  Towards Availability Benchmarks: A Case Study of Software RAID Systems , 2000, USENIX Annual Technical Conference, General Track.

[30]  Marios D. Dikaiakos,et al.  Characterization of Computational Grid Resources Using Low-Level Benchmarks , 2006, 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06).

[31]  David A. Patterson,et al.  Practical Issues in Dependability Benchmarking , 2002 .

[32]  Jennifer M. Schopf,et al.  A performance study of monitoring and information services for distributed systems , 2003, High Performance Distributed Computing, 2003. Proceedings. 12th IEEE International Symposium on.

[33]  Elisa Heymann,et al.  The EU-CrossGrid Approach for Grid Application Scheduling , 2003, European Across Grids Conference.

[34]  Anthony J. G. Hey,et al.  The role of performance engineering techniques in the context of the Grid , 2005, Concurr. Pract. Exp..

[35]  Marios D. Dikaiakos,et al.  First Prototype of the CrossGrid Testbed , 2003, European Across Grids Conference.

[36]  Karsten Schwan,et al.  Open Metadata Formats: Efficient XML-Based Communication for High Performance Computing , 2004, Cluster Computing.

[37]  David A. Patterson,et al.  A Simple Way to Estimate the Cost of Downtime , 2002, LISA.

[38]  Douglas Thain,et al.  Building Reliable Clients and Services , 2004, The Grid 2, 2nd Edition.

[39]  Marian Bubak,et al.  The CrossGrid Architecture: Applications, Tools, and Grid Services , 2003, European Across Grids Conference.

[40]  Henri Casanova,et al.  Benchmark probes for grid assessment , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..