Falkon: a Fast and Light-weight tasK executiON framework

To enable the rapid execution of many tasks on compute clusters, we have developed Falkon, a Fast and Light-weight tasK executiON framework. Falkon integrates (1) multi-level scheduling to separate resource acquisition (via, e.g., requests to batch schedulers) from task dispatch, and (2) a streamlined dispatcher. Falkon's integration of multi-level scheduling and streamlined dispatchers delivers performance not provided by any other system. We describe Falkon architecture and implementation, and present performance results for both microbenchmarks and applications. Microbenchmarks show that Falkon throughput (487 tasks/sec) and scalability (to 54,000 executors and 2,000,000 tasks processed in just 112 minutes) are one to two orders of magnitude better than other systems used in production Grids. Large-scale astronomy and medical applications executed under Falkon by the Swift parallel programming system achieve up to 90% reduction in end-to-end run time, relative to versions that execute tasks via separate scheduler submissions.

[1]  Flaviu Cristian,et al.  A Highly Available Local Leader Election Service , 1999, IEEE Trans. Software Eng..

[2]  Nancy A. Lynch,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[3]  Benny Rochwerger,et al.  Oceano-SLA based management of a computing utility , 2001, 2001 IEEE/IFIP International Symposium on Integrated Network Management Proceedings. Integrated Network Management VII. Integrated Management Strategies for the New Millennium (Cat. No.01EX470).

[4]  Yul-Wan Sung,et al.  Functional magnetic resonance imaging , 2004, Scholarpedia.

[5]  Krithi Ramamritham,et al.  The Spring System: Integrated Support for Complex Real-Time Systems , 1999, Real-Time Systems.

[6]  Wolfgang Gentzsch,et al.  Sun Grid Engine: towards creating a compute power grid , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.

[7]  Rajeev Motwani,et al.  Estimating Aggregates on a Peer-to-Peer Network , 2003 .

[8]  Danny Dolev,et al.  On the minimal synchronism needed for distributed consensus , 1983, 24th Annual Symposium on Foundations of Computer Science (sfcs 1983).

[9]  Ian Foster,et al.  AstroPortal: A Science Gateway for Large-scale Astronomy Data Analysis , 2006 .

[10]  Von Welch.  Globus toolkit version 4 grid security infras-tructur: A standards perspective , 2004 .

[11]  Gregor von Laszewski,et al.  Swift: Fast, Reliable, Loosely Coupled Parallel Computation , 2007, 2007 IEEE Congress on Services (Services 2007).

[12]  L. Ramakrishnan,et al.  Toward a Doctrine of Containment: Grid Hosting with Adaptive Resource Control , 2006, ACM/IEEE SC 2006 Conference (SC'06).

[13]  Avishai Wool,et al.  Probabilistic Quorum Systems , 2001, Inf. Comput..

[14]  Zhou Lei,et al.  The portable batch scheduler and the maui scheduler on linux clusters , 2000 .

[15]  Douglas Thain,et al.  Distributed computing in practice: the Condor experience , 2005, Concurr. Pract. Exp..

[16]  Christos Gkantsidis,et al.  Random walks in peer-to-peer networks , 2004, IEEE INFOCOM 2004.

[17]  Daniel S. Katz,et al.  Pegasus: A framework for mapping complex scientific workflows onto distributed systems , 2005, Sci. Program..

[18]  Yong Zhao,et al.  Chimera: a virtual data system for representing, querying, and automating data derivation , 2002, Proceedings 14th International Conference on Scientific and Statistical Database Management.

[19]  Andrew S. Tanenbaum,et al.  Distributed systems: Principles and Paradigms , 2001 .

[20]  Gerard Tel,et al.  Introduction to Distributed Algorithms: Contents , 2000 .

[21]  Indranil Gupta,et al.  A Probabilistically Correct Leader Election Protocol for Large Groups , 2000, DISC.

[22]  Jeff T. Linderoth,et al.  An enabling framework for master-worker applications on the Computational Grid , 2000, Proceedings the Ninth International Symposium on High-Performance Distributed Computing.

[23]  Alexandru Iosup,et al.  The Characteristics and Performance of Groups of Jobs in Grids , 2007, Euro-Par.

[24]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM 2001.

[25]  Steven Tuecke,et al.  The Anatomy of the Grid , 2003 .

[26]  David R. Karger,et al.  Chord: a scalable peer-to-peer lookup protocol for internet applications , 2003, TNET.

[27]  Nancy A. Lynch,et al.  Electing a leader in a synchronous ring , 1987, JACM.

[28]  Ian T. Foster.  Globus Toolkit Version 4: Software for Service-Oriented Systems , 2005, NPC.

[29]  Alexander S. Szalay,et al.  Harnessing grid resources to enable the dynamic analysis of large astronomy datasets , 2006, SC.

[30]  D. Katz,et al.  The Montage architecture for grid-enabled science processing of large, distributed datasets , 2004 .

[31]  Dahlia Malkhi,et al.  Estimating network size from local information , 2003, Information Processing Letters.

[32]  C. Kesselman,et al.  Performance Impact of Resource Provisioning on Workflows , 2005 .

[33]  David P. Anderson,et al.  BOINC: a system for public-resource computing and storage , 2004, Fifth IEEE/ACM International Workshop on Grid Computing.

[34]  Edward Walker,et al.  Creating personal adaptive clusters for managing scientific jobs in a distributed computing environment , 2006, 2006 IEEE Challenges of Large Applications in Distributed Environments.

[35]  Daniel S. Katz,et al.  Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand , 2004, SPIE Astronomical Telescopes + Instrumentation.

[36]  Ömer Egecioglu,et al.  Billiard Quorums on the Grid , 1997, Inf. Process. Lett..

[37]  Suresh Jagannathan,et al.  Randomized Protocols for Duplicate Elimination in Peer-to-Peer Storage Systems , 2005, Peer-to-Peer Computing.

[38]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[39]  Ian Foster,et al.  GT4 GRAM: A Functionality and Performance Study , 2007 .

[40]  Mamoru Maekawa,et al.  A N algorithm for mutual exclusion in decentralized systems , 1985, TOCS.

[41]  Yuh-Jzer Joung,et al.  Asynchronous group mutual exclusion , 2000, Distributed Computing.

[42]  Kavitha Ranganathan,et al.  Simulation Studies of Computation and Data Scheduling Algorithms for Data Grids , 2003, Journal of Grid Computing.

[43]  Ian T. Foster,et al.  The Anatomy of the Grid: Enabling Scalable Virtual Organizations , 2001, Int. J. High Perform. Comput. Appl..

[44]  Alexandru Iosup,et al.  How are Real Grids Used? The Analysis of Four Grid Traces and Its Implications , 2006, 2006 7th IEEE/ACM International Conference on Grid Computing.

[45]  Ian T. Foster,et al.  Condor-G: A Computation Management Agent for Multi-Institutional Grids , 2004, Cluster Computing.

[46]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[47]  Alexander S. Szalay,et al.  The importance of data locality in distributed computing applications , 2006 .

[48]  Carl Kesselman,et al.  Optimizing Grid-Based Workflow Execution , 2005, Journal of Grid Computing.

[49]  Nancy Wilkins-Diehr,et al.  TeraGrid: Analysis of Organization, System Architecture, and Middleware Enabling New Types of Applications , 2006, High Performance Computing Workshop.

[50]  David P. Anderson,et al.  High-performance task distribution for volunteer computing , 2005, First International Conference on e-Science and Grid Computing (e-Science'05).

[51]  Yong Zhao,et al.  Dynamic Resource Provisioning in Grid Environments , 2007 .

[52]  Stephen P. Boyd,et al.  Fastest Mixing Markov Chain on a Graph , 2004, SIAM Rev..

[53]  W. Szpankowski.  Average Case Analysis of Algorithms on Sequences , 2001 .

[54]  Eyal Kushilevitz,et al.  Randomized mutual exclusion algorithms revisited , 1992, PODC '92.

[55]  Jared Saia,et al.  Choosing a random peer , 2004, PODC '04.

[56]  David J. DeWitt,et al.  Turning Cluster Management into Data Management; A System Overview , 2006, CIDR.

[57]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[58]  Miron Livny,et al.  Condor and the Grid , 2003 .

[59]  John E. Hershey,et al.  Computation , 1991, Digit. Signal Process..

[60]  Israel Cidon,et al.  Propagation and Leader Election in a Multihop Broadcast Environment , 1998, DISC.

[61]  Peter Druschel,et al.  Resource containers: a new facility for resource management in server systems , 1999, OSDI '99.