Comparative study between Hadoop and Spark based on Hibench benchmarks

Big Data is currently a hot topic for companies and scientists around the world, due to the emergence of new technologies, devices and communication means like social network sites, which led to a noticeable increase of the amount of data produced every year, even every day. In addition, traditional algorithms and technologies are inefficient to process, analyze and store this vast amount of data. So, to solve this problem, Big Data frameworks are needed. In this paper, we present and discuss a performance comparison between two popular Big Data frameworks. Hadoop and Spark, which are used to efficiently process vast amount of data in parallel and distributed mode on a large clusters. Hibench benchmark suite is used to compare the performance of these two frameworks based on the criteria as execution time, throughput and speedup. Our experimental results show that Spark is more efficient than Hadoop to deal with large amount of data. However, spark requires higher memory allocation, since it loads processes into memory and keeps them in caches for a while, just like standard databases. So the choice depends on performance level and memory constraints.