Nebu: A topology-aware deployment system for reliable virtualized multi-cluster environments

Petabytes of data are processed daily by distributed applications built upon Hadoop and MongoDB. A significant fraction of these applications use cloud infrastructure to cope with this vast amount of data. Commercial clouds use virtualized environments, but most distributed applications are designed around the idea that they run on physical hardware. When this is no longer the case, guarantees for an application’s reliability and performance no longer hold. To remedy this issue, we design a powerful and comprehensive system called Nebu. Nebu is able to provide information about the physical topology of the cloud to the distributed application and is capable of automated virtual machine and application deployment. Nebu performs these tasks without depending on any single distributed application or virtual machine manager. Instead, Nebu provides efficient APIs that make it easy to provide compatibility with many popular distributed applications and virtual machine managers. We develop Nebu as an open source project using modern software engineering practices. In particular, we use the agile development method Scrum in combination with the Kanban scheduling system. We apply iterative API design through the use of RAML and supporting UML diagrams. Because no effective methods for testing distributed applications have been developed, we use both unit testing and manual testing. We also apply automated regression testing through the use of continuous integration. Because there are no formal guidelines on how to validate distributed applications for the kind we investigate in this work, we develop Nebu and perform real-world experiments with multiple distributed applications using an enterprise multi-cluster infrastructure. These experiments show that Nebu enables applications to give guarantees about reliability without degrading their performance. To increase Nebu’s usability, we provide extensions that offer compatibility with the distributed applications Hadoop and MongoDB, and virtual machine manager VMware. Both the system and the extensions to the system are developed in an enterprise environment. This holds good promise that Nebu will be adopted by open-source communities, as well as the industry.