Near-Optimal Distributed Band-Joins through Recursive Partitioning

We consider running-time optimization for band-joins in a distributed system, e.g., the cloud. To balance load across worker machines, input has to be partitioned, which causes duplication. We explore how to resolve this tension between maximum load per worker and input duplication for band-joins between two relations. Previous work suffered from high optimization cost or considered partitionings that were too restricted (resulting in suboptimal join performance). Our main insight is that recursive partitioning of the join-attribute space with the appropriate split scoring measure can achieve both low optimization cost and low join cost. It is the first approach that is not only effective for one-dimensional band-joins but also for joins on multiple attributes. Experiments indicate that our method is able to find partitionings that are within 10% of the lower bound for both maximum load per worker and input duplication for a broad range of settings, significantly improving over previous work.

[1]  Masaru Kitsuregawa,et al.  Bucket Spreading Parallel Hash: A New, Robust, Parallel Hash Join Method for Data Skew in the Super Database Computer (SDC) , 1990, VLDB.

[2]  Alfred G. Dale,et al.  A Taxonomy and Performance Model of Data Skew Effects in Parallel Joins , 1991, VLDB.

[3]  Yeye He,et al.  ClusterJoin: A Similarity Joins Framework using Map-Reduce , 2014, Proc. VLDB Endow..

[4]  David J. DeWitt,et al.  Practical Skew Handling in Parallel Joins , 1992, VLDB.

[5]  Masaru Kitsuregawa,et al.  Dynamic Join Product Skew Handling for Hash-Joins in Shared-Nothing Database Systems , 1995, DASFAA.

[6]  Jignesh M. Patel,et al.  A comparison of join algorithms for log processing in MaPreduce , 2010, SIGMOD Conference.

[7]  Jian Pei,et al.  Data Mining : Concepts and Techniques 3rd edition Ed. 3 , 2011 .

[8]  Kesheng Wu,et al.  Similarity Join over Array Data , 2016, SIGMOD Conference.

[9]  C. J. Hahn,et al.  Extended Edited Synoptic Cloud Reports from Ships and Land Stations Over the Globe, 1952-1996 , 1999 .

[10]  Valery Soloviev.  A truncating hash algorithm for processing band-join queries , 1993, Proceedings of IEEE 9th International Conference on Data Engineering.

[11]  Philip S. Yu,et al.  New Algorithms for Parallelizing Relational Database Joins in the Presence of Data Skew , 1994, IEEE Trans. Knowl. Data Eng..

[12]  Min Wang,et al.  Efficient Multi-way Theta-Join Processing Using MapReduce , 2012, Proc. VLDB Endow..

[13]  Alfons Kemper,et al.  Flow-Join: Adaptive skew handling for distributed joins over high-speed networks , 2016, 2016 IEEE 32nd International Conference on Data Engineering (ICDE).

[14]  Carsten Binnig,et al.  The End of Slow Networks: It's Time for a Redesign , 2015, Proc. VLDB Endow..

[15]  Dan Suciu,et al.  Skew in parallel query processing , 2014, PODS.

[16]  Peter Baumann,et al.  Storage of multidimensional arrays based on arbitrary tiling , 1999, Proceedings 15th International Conference on Data Engineering (Cat. No.99CB36337).

[17]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[18]  Peter J. Haas,et al.  Improved histograms for selectivity estimation of range predicates , 1996, SIGMOD '96.

[19]  Christoph Koch,et al.  Load balancing and skew resilience for parallel joins , 2016, 2016 IEEE 32nd International Conference on Data Engineering (ICDE).

[20]  David J. DeWitt,et al.  Multiprocessor Hash-Based Join Algorithms , 1985, VLDB.

[21]  Ulf Leser,et al.  Set Similarity Joins on MapReduce: An Experimental Survey , 2018, Proc. VLDB Endow..

[22]  Dan Suciu,et al.  From Theory to Practice: Efficient Join Query Evaluation in a Parallel Database System , 2015, SIGMOD Conference.

[23]  Yannis E. Ioannidis,et al.  Selectivity Estimation Without the Attribute Value Independence Assumption , 1997, VLDB.

[24]  Kien A. Hua,et al.  Handling Data Skew in Multiprocessor Database Computers Using Partition Tuning , 1991, VLDB.

[25]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[26]  Salvatore J. Stolfo,et al.  Predictive dynamic load balancing of parallel hash-joins over heterogeneous processors in the presence of data skew , 1994, Proceedings of 3rd International Conference on Parallel and Distributed Information Systems.

[27]  David J. DeWitt,et al.  A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment , 1989, SIGMOD '89.

[28]  Hongjun Lu,et al.  Load Balanced Join Processing in Shared-Noting Systems , 1994, J. Parallel Distributed Comput..

[29]  Paolo Papotti,et al.  Fast and scalable inequality joins , 2017, The VLDB Journal.

[30]  David J. DeWitt,et al.  Tradeoffs in Processing Complex Join Queries via Hashing in Multiprocessor Database Machines , 1990, VLDB.

[31]  Yannis E. Ioannidis,et al.  Estimation of Query-Result Distribution and its Application in Parallel-Join Load Balancing , 1996, VLDB.

[32]  Hidehiko Tanaka,et al.  Application of hash to data base machine and its architecture , 1983, New Generation Computing.

[33]  Michael Stonebraker,et al.  Skew-Aware Join Optimization for Array Databases , 2015, SIGMOD Conference.

[34]  David J. DeWitt,et al.  An Evaluation of Non-Equijoin Algorithms , 1991, VLDB.

[35]  Xinyan Deng,et al.  Submodularity of Distributed Join Computation , 2018, SIGMOD Conference.

[36]  Liang Chen,et al.  Handling data skew in parallel joins in shared-nothing systems , 2008, SIGMOD Conference.

[37]  Nicolas Bruno,et al.  Advanced Join Strategies for Large-Scale Distributed Computation , 2014, Proc. VLDB Endow..

[38]  C. J. Hahn,et al.  Extended edited synoptic cloud reports from ships and land stations over the globe , 1999 .

[39]  Christoph Koch,et al.  Scalable and Adaptive Online Joins , 2014, Proc. VLDB Endow..

[40]  Kenneth A. Ross,et al.  Track join: distributed joins with minimal network traffic , 2014, SIGMOD Conference.

[41]  Yizhou Sun,et al.  Abstract cost models for distributed data-intensive computations , 2018, Distributed and Parallel Databases.

[42]  Anastasios Gounaris,et al.  Flexible partitioning for selective binary theta-joins in a massively parallel setting , 2018, Distributed and Parallel Databases.

[43]  Mirek Riedewald,et al.  Processing theta-joins using MapReduce , 2011, SIGMOD '11.