Distributed data systems systems are used in a variety of settings like online serving, offline analytics, data transport, and search, among other use cases. They let organizations scale out their workloads using cost-effective commodity hardware, while retaining key properties like fault tolerance and scalability. At LinkedIn we have built a number of such systems. A key pattern we observe is that even though they may serve different purposes, they tend to have a lot of common functionality, and tend to use common building blocks in their architectures. One such building block that is just beginning to receive attention is cluster management, which addresses the complexity of handling a dynamic, large-scale system with many servers. Such systems must handle software and hardware failures, setup tasks such as bootstrapping data, and operational issues such as data placement, load balancing, planned upgrades, and cluster expansion.
All of this shared complexity, which we see in all of our systems, motivates us to build a cluster management framework, Helix, to solve these problems once in a general way.
Helix provides an abstraction for a system developer to separate coordination and management tasks from component functional tasks of a distributed system. The developer defines the system behavior via a state model that enumerates the possible states of each component, the transitions between those states, and constraints that govern the system's valid settings. Helix does the heavy lifting of ensuring the system satisfies that state model in the distributed setting, while also meeting the system's goals on load balancing and throttling state changes. We detail several Helix-managed production distributed systems at LinkedIn and how Helix has helped them avoid building custom management components. We describe the Helix design and implementation and present an experimental study that demonstrates its performance and functionality.
[1]
Sanjay Ghemawat,et al.
MapReduce: Simplified Data Processing on Large Clusters
,
2004,
OSDI.
[2]
Robbert van Renesse,et al.
Experiences with the Amoeba distributed operating system
,
1990,
CACM.
[3]
Ethan L. Miller,et al.
Replication under scalable hashing: a family of algorithms for scalable decentralized data distribution
,
2004,
18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..
[4]
Hans-Arno Jacobsen,et al.
PNUTS: Yahoo!'s hosted data serving platform
,
2008,
Proc. VLDB Endow..
[5]
Xin Chen,et al.
F1: the fault-tolerant distributed RDBMS supporting google's ad business
,
2012,
SIGMOD Conference.
[6]
Randy H. Katz,et al.
The Datacenter Needs an Operating System
,
2011,
HotCloud.
[7]
Wilson C. Hsieh,et al.
Bigtable: A Distributed Storage System for Structured Data
,
2006,
TOCS.
[8]
John F. Karpovich,et al.
Resource management in Legion
,
1999,
Future Gener. Comput. Syst..
[9]
Amin Vahdat,et al.
GLUix: a global layer unix for a network of workstations
,
1998
.
[10]
Ian T. Foster,et al.
Condor-G: A Computation Management Agent for Multi-Institutional Grids
,
2004,
Cluster Computing.
[11]
Lei Gao,et al.
Data Infrastructure at LinkedIn
,
2012,
2012 IEEE 28th International Conference on Data Engineering.