Automated Spark Clusters Deployment for Big Data with Standalone Applications Integration

The huge amount of data stored nowadays has turned big data analytics into a very trendy research field. Spark has emerged as a very powerful and widely used paradigm for clusters deployment and big data management. However, to get started is still a very tough task, due to the excessive requisites that all nodes must fulfil. Thus, this work introduces a web service specifically designed for an easy and efficient Spark cluster management. In particular, a service with a friendly graphical user interface has been developed to automate the deploying of clusters. Another relevant feature is the possibility of integrating any algorithm into the web service. That is, the user only needs to provide the executable file and the number of required inputs for a proper parametrization. Finally, an illustrative case study is included to show ad hoc algorithms usage (the MLlib implementation for k-means, in this case) across the nodes of the configured cluster.