Auto Tuning of Hadoop and Spark parameters

Data of the order of terabytes, petabytes, or beyond is known as Big Data. This data cannot be processed using the traditional database software, and hence there comes the need for Big Data Platforms. By combining the capabilities and features of various big data applications and utilities, Big Data Platforms form a single solution. It is a platform that helps to develop, deploy and manage the big data environment. Hadoop and Spark are the two open-source Big Data Platforms provided by Apache. Both these platforms have many configurational parameters, which can have unforeseen effects on the execution time, accuracy, etc. Manual tuning of these parameters can be tiresome, and hence automatic ways should be needed to tune them. After studying and analyzing various previous works in automating the tuning of these parameters, this paper proposes two algorithms Grid Search with Finer Tuning and Controlled Random Search. The performance indicator studied in this paper is Execution Time. These algorithms help to tune the parameters automatically. Experimental results have shown a reduction in execution time of about 70% and 50% for Hadoop and 81.19% and 77.77% for Spark by Grid Search with Finer Tuning and Controlled Random Search, respectively.

[1]  Yi Wang,et al.  Otterman: A Novel Approach of Spark Auto-tuning by a Hybrid Strategy , 2018, 2018 5th International Conference on Systems and Informatics (ICSAI).

[2]  J. Hossen,et al.  SMBSP: A Self-Tuning Approach using Machine Learning to Improve Performance of Spark in Big Data Processing , 2018, 2018 7th International Conference on Computer and Communication Engineering (ICCCE).

[3]  Nhan Nguyen,et al.  Towards Automatic Tuning of Apache Spark Configuration , 2018, 2018 IEEE 11th International Conference on Cloud Computing (CLOUD).

[4]  Wyn L. Price,et al.  A Controlled Random Search Procedure for Global Optimisation , 1977, Comput. J..

[5]  Nando de Freitas,et al.  Bayesian Optimization in a Billion Dimensions via Random Embeddings , 2013, J. Artif. Intell. Res..

[6]  Chih-Jen Lin,et al.  A Practical Guide to Support Vector Classication , 2008 .

[7]  Jing Gu,et al.  Auto-Tuning Spark Configurations Based on Neural Network , 2018, 2018 IEEE International Conference on Communications (ICC).

[8]  Yi Liu,et al.  JellyFish: Online Performance Tuning with Adaptive Configuration and Elastic Container in Hadoop Yarn , 2015, 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS).

[9]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[10]  Manish Singhal,et al.  A Framework for Performance Analysis and Tuning in Hadoop Based Clusters , 2014 .

[11]  Sayali Ashok Shivarkar.  Speed-up Extension to Hadoop System , 2014 .

[12]  Dick H. J. Epema,et al.  Towards Machine Learning-Based Auto-tuning of MapReduce , 2013, 2013 IEEE 21st International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems.