Cluster Performance Simulation for Spark Deployment Planning, Evaluation and Optimization

As the most active project in the Hadoop ecosystem these days [1], Spark is a fast and general purpose engine for large-scale data processing. Spark runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk [2]. However, Spark performance is impacted by many factors especially memory and JVM related, which makes capacity planning and tuning for Spark clusters extremely difficult. Current estimation based solution are highly dependent on experience which are trial-and-error and far from efficient and accurate. Here, we propose a novel Spark simulator based on CSMethod [3], extension with a fine-grained multi-layered memory subsystem, well suitable for this scenario. The whole Spark application execution life cycle is simulated, hardware activities derived from software operations are dynamically mapped onto architecture models for processors, storage, and network devices. Experimental results with several popular micro benchmarks and a real case IoT workloads demonstrate that our Spark Simulator achieves high accuracy with an average error rate below 7%, with light weight computing resource. Case studies are also demonstrated to show the simulator’s capability.