DistGear: A Lightweight Event-Driven Framework for Developing Distributed Applications

The emergence of cloud computing makes it easier for enterprises and individual users to obtain distributed computing resources. At the same time, developing distributed systems is still complex and difficult. In order to reduce the complexity in distributed application development, a series of techniques such as RPC, message-oriented middleware have been proposed and widely used. These techniques hide the underlying communicating details but require developers to take control of coordination among different machines for complex distributed tasks. This paper proposes a lightweight event-driven framework called DistGear to simplify the development of complex task processing in distributed applications. DistGear is a layer of software built on top of communication facilities for providing developers with a programming abstraction to easily handle distributed tasks and control machine coordination in tasks. We implement DistGear based on coroutines and message communication. Experience with DistGear in a simplified real-world system suggests it is effective. Performance evaluation also demonstrates DistGear's good concurrency and low resources consumption.

[1]  Eddie Kohler,et al.  Events Can Make Sense , 2007, USENIX Annual Technical Conference.

[2]  Pieter Hintjens,et al.  ZeroMQ: Messaging for Many Applications , 2013 .

[3]  Daniel P. Friedman,et al.  Aspects of Applicative Programming for Parallel Processing , 1978, IEEE Transactions on Computers.

[4]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[5]  Peter R. Pietzuch,et al.  Hermes: a distributed event-based middleware architecture , 2002, Proceedings 22nd International Conference on Distributed Computing Systems Workshops.

[6]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[7]  Randy H. Katz,et al.  Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center , 2011, NSDI.

[8]  Muneeb Ali,et al.  Protothreads: simplifying event-driven programming of memory-constrained embedded systems , 2006, SenSys '06.

[9]  Jignesh M. Patel,et al.  Storm@twitter , 2014, SIGMOD Conference.

[10]  日経BP社,et al.  Amazon Web Services完全ソリューションガイド , 2016 .

[11]  Andrew Birrell,et al.  Implementing remote procedure calls , 1984, TOCS.

[12]  Carl Hewitt,et al.  A Universal Modular ACTOR Formalism for Artificial Intelligence , 1973, IJCAI.

[13]  Peter R. Pietzuch,et al.  Distributed event-based systems , 2006 .

[14]  Alfonso Fuggetta,et al.  The JEDI Event-Based Infrastructure and Its Application to the Development of the OPSS WFMS , 2001, IEEE Trans. Software Eng..

[15]  Bo An,et al.  Workspace as a Service: An Online Working Environment for Private Cloud , 2017, 2017 IEEE Symposium on Service-Oriented System Engineering (SOSE).

[16]  Kenneth P. Birman,et al.  Exploiting virtual synchrony in distributed systems , 1987, SOSP '87.

[17]  Carl Hewitt,et al.  The incremental garbage collection of processes , 1977, Artificial Intelligence and Programming Languages.