This paper presents Polybase, a feature of SQL Server PDW V2 that allows users to manage and query data stored in a Hadoop cluster using the standard SQL query language. Unlike other database systems that provide only a relational view over HDFS-resident data through the use of an external table mechanism, Polybase employs a split query processing paradigm in which SQL operators on HDFS-resident data are translated into MapReduce jobs by the PDW query optimizer and then executed on the Hadoop cluster. The paper describes the design and implementation of Polybase along with a thorough performance evaluation that explores the benefits of employing a split query processing paradigm for executing queries that involve both structured data in a relational DBMS and unstructured data in Hadoop. Our results demonstrate that while the use of a split-based query execution paradigm can improve the performance of some queries by as much as 10X, one must employ a cost-based query optimizer that considers a broad set of factors when deciding whether or not it is advantageous to push a SQL operator to Hadoop. These factors include the selectivity factor of the predicate, the relative sizes of the two clusters, and whether or not their nodes are co-located. In addition, differences in the semantics of the Java and SQL languages must be carefully considered in order to avoid altering the expected results of a query.
[1]
Beng Chin Ooi,et al.
Query optimization for massively parallel data processing
,
2011,
SoCC.
[2]
Abraham Silberschatz,et al.
Efficient processing of data warehousing queries in a split execution environment
,
2011,
SIGMOD '11.
[3]
Herodotos Herodotou,et al.
Profiling, what-if analysis, and cost-based optimization of MapReduce programs
,
2011,
Proc. VLDB Endow..
[4]
Yu Xu,et al.
Integrating hadoop and parallel DBMs
,
2010,
SIGMOD Conference.
[5]
Yu Xu,et al.
A Hadoop based distributed loading approach to parallel data warehouses
,
2011,
SIGMOD '11.
[6]
David J. DeWitt,et al.
Query optimization in microsoft SQL server PDW
,
2012,
SIGMOD Conference.