An In-depth Investigation of Large-scale RDF Relational Schema Optimizations Using Spark-SQL

This paper discusses one of the most significant challenges of large-scale RDF data processing over Apache Spark, the relational schema optimization. The choice of RDF partitioning techniques and storage formats using SparkSQL significantly impacts query performance. The impact of the relational schemas and the underlying data storage formats is indisputable; they significantly affect the query performance. Nevertheless, the trade-offs in different configurations have not been a subject of intensive study in the literature. This paper presents an in-depth investigation for practitioners to understand such trade-offs and their best practices. It also reports on the pitfalls behind the implementation SPARQL optimizations over SparkSQL. Our experiments provide insights into these schemas’ relative strengths by comparing three different partitioning techniques and four other storage formats. Our results draw a better understanding of the current State-Of-The-Art (S.O.T.A) and pave the way for a wide range of best practices and systematically tuning the performance of distributed systems to handle vast RDF data.