Embedded Gossip: Lightweight Online Measurement for Large-Scale Applications

For large-scale parallel applications, lightweight online monitoring can enable a wide range of online adaptations, including load balancing, power management, and progress monitoring. The processing and monitoring overhead of centralized global tracing techniques make them unsuitable for such tasks. Purely local tools, on the other hand, fail to provide the global information necessary for many desirable online adaptations of large-scale applications. In this paper, we describe a novel distributed online measurement method for large-scale applications called Embedded Gossip (EG). EG works by piggybacking performance information about application behavior on existing application messages and merging received information with previously known data in a fashion customized to the needs of a particular monitoring task. EG thus provides each process with both local and global views of application behavior with low overhead. To illustrate the capabilities of Embedded Gossip, we also show that it disseminates global information in a timely fashion for a wide range of monitoring tasks, including critical path profiling, workload imbalance monitoring, and progress monitoring. This global information has a wide range of potential uses, including imbalance detection for load balancing and energy management tools, progress monitoring for batch schedulers, and a wide range of other performance debugging and optimization techniques.

[1]  Ben H. H. Juurlink,et al.  Gossiping on Meshes and Tori , 1998, IEEE Trans. Parallel Distributed Syst..

[2]  Diana Marculescu,et al.  Microarchitecture-level power management , 2002, IEEE Trans. Very Large Scale Integr. Syst..

[3]  David E. Culler,et al.  The ganglia distributed monitoring system: design, implementation, and experience , 2004, Parallel Comput..

[4]  Allen D. Malony,et al.  TAUg: Runtime Global Performance Data Access Using MPI , 2006, PVM/MPI.

[5]  Jeffrey K. Hollingsworth An online computation of critical path profiling , 1996, SPDT '96.

[6]  V. Springel,et al.  GADGET: a code for collisionless and gasdynamical cosmological simulations , 2000, astro-ph/0003162.

[7]  David F. Heidel,et al.  An Overview of the BlueGene/L Supercomputer , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[8]  Pradip K. Srimani,et al.  An efficient distributed protocol for online gossiping problem , 2005, IEEE/WIC/ACM International Conference on Intelligent Agent Technology.

[9]  Richard Mortier,et al.  Magpie: Online Modelling and Performance-aware Systems , 2003, HotOS.

[10]  Maurice Yarrow,et al.  New Implementations and Results for the NAS Parallel Benchmarks 2 , 1997, PPSC.

[11]  Message Passing Interface Forum MPI: A message - passing interface standard , 1994 .

[12]  Garrick Staples,et al.  TORQUE resource manager , 2006, SC.

[13]  Allen D. Malony,et al.  Portable profiling and tracing for parallel, scientific applications using C++ , 1998, SPDT '98.

[14]  Marcos K. Aguilera,et al.  Performance debugging for distributed systems of black boxes , 2003, SOSP '03.

[15]  Courtenay T. Vaughan,et al.  Design of dynamic load-balancing tools for parallel applications , 2000, ICS '00.

[16]  Christian Poellabauer,et al.  dproc - Extensible Run-Time Resource Monitoring for Cluster Applications , 2002, International Conference on Computational Science.

[17]  D.A. Reed,et al.  Scalable performance analysis: the Pablo performance analysis environment , 1993, Proceedings of Scalable Parallel Libraries Conference.

[18]  William F. Mitchell,et al.  Optimal Multilevel Iterative Methods for Adaptive Grids , 1992, SIAM J. Sci. Comput..

[19]  Arthur B. Maccabe,et al.  Online Critical Path Profiling for Parallel Applications , 2005, 2005 IEEE International Conference on Cluster Computing.

[20]  Stéphane Pérennes,et al.  Circuit-Switched Gossiping in 3-Dimensional Torus Networks , 1996, Euro-Par, Vol. I.

[21]  Ying Zhang,et al.  SvPablo: A Multi-language Performance Analysis System , 1998, Computer Performance Evaluation.

[22]  Barton P. Miller,et al.  The Paradyn Parallel Performance Measurement Tool , 1995, Computer.

[23]  Alan L. Cox,et al.  Causeway: Operating System Support for Controlling and Analyzing the Execution of Distributed Programs , 2005, HotOS.

[24]  Jack J. Dongarra,et al.  A Portable Programming Interface for Performance Evaluation on Modern Processors , 2000, Int. J. High Perform. Comput. Appl..

[25]  Ulrich Meyer,et al.  Oblivious Gossiping on Tori , 2002, J. Algorithms.

[26]  Ronald Minnich,et al.  Supermon: a high-speed cluster monitoring system , 2002, Proceedings. IEEE International Conference on Cluster Computing.

[27]  José E. Moreira,et al.  Blue Gene/L performance tools , 2005, IBM J. Res. Dev..

[28]  Barton P. Miller,et al.  Critical path analysis for the execution of parallel and distributed programs , 1988, [1988] Proceedings. The 8th International Conference on Distributed.

[29]  Konrad Slind,et al.  Monitoring distributed systems , 1987, TOCS.

[30]  Michal Soch,et al.  Optimal Gossip in Store-and-Forward Noncombining 2-D Tori , 1997, Euro-Par.

[31]  Jeffrey S. Vetter,et al.  Dynamic statistical profiling of communication activity in distributed applications , 2002, SIGMETRICS '02.

[32]  Jeffrey S. Vetter,et al.  Autopilot: adaptive control of distributed applications , 1998, Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244).

[33]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[34]  G. Bryan,et al.  Introducing Enzo, an AMR Cosmology Application , 2004, astro-ph/0403044.

[35]  William E. Johnston,et al.  The NetLogger methodology for high performance distributed systems performance analysis , 1998, Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244).

[36]  J. C. Yan,et al.  Performance tuning with AIMS/spl minus/an Automated Instrumentation and Monitoring System for multicomputers , 1994, 1994 Proceedings of the Twenty-Seventh Hawaii International Conference on System Sciences.

[37]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[38]  Bernd Mohr,et al.  TAU: A Portable Parallel Program Analysis Environment for pC++ , 1994, CONPAR.

[39]  27th International Conference on Distributed Computing Systems Workshops (ICDCS 2007 Workshops), June 25-29, 2007, Toronto, Ontario, Canada , 2007, ICDCS Workshops.

[40]  Jerry C. Yan Performance Tuning with AIMS - An Automated Instrumentation and Monitoring System for Multicomputers , 1994, HICSS.

[41]  Lorenzo Alvisi,et al.  Causality tracking in causal message-logging protocols , 2002, Distributed Computing.