FreshJoin: An Efficient and Adaptive Algorithm for Set Containment Join

This paper revisits set containment join (SCJ) problem, which uses the subset relationship (i.e., ⊆\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\subseteq$$\end{document}) as condition to join set-valued attributes of two relations and has many fundamental applications in commercial and scientific fields. Existing in-memory algorithms for SCJ are either signature-based or prefix-tree-based. The former incurs high CPU cost because of the enumeration of signatures, while the latter incurs high space cost because of the storage of prefix trees. This paper proposes a new adaptive parameter-free in-memory algorithm, named as frequency-hashjoin or FreshJoin\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathsf {FreshJoin}}$$\end{document} in short, to evaluate SCJ efficiently. FreshJoin\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathsf {FreshJoin}}$$\end{document} builds a flat index on-the-fly to record three kinds of signatures (i.e., two least frequent elements and a hash signature whose length is determined adaptively by the frequencies of elements in the universe set). The index consists of two sparse inverted indices and two arrays which record hash signatures of all sets in each relation. The index is well organized such that FreshJoin\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathsf {FreshJoin}}$$\end{document} can avoid enumerating hash signatures. The rationality of this design is explained. And, the time and space cost of the proposed algorithm, which provide a rule to choose FreshJoin\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathsf {FreshJoin}}$$\end{document} from existing algorithms, are analyzed. Experiments on 16 real-life datasets show that FreshJoin\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathsf {FreshJoin}}$$\end{document} usually reduces more than 50% of space cost while remains as competitive as the state-of-the-art algorithms in running time.

[1]  Surajit Chaudhuri,et al.  A Primitive Operator for Similarity Joins in Data Cleaning , 2006, 22nd International Conference on Data Engineering (ICDE'06).

[2]  Hong-zhi Wang,et al.  FrepJoin: an efficient partition-based algorithm for edit similarity join , 2017, Frontiers of Information Technology & Electronic Engineering.

[3]  Roberto J. Bayardo,et al.  Scaling up all pairs similarity search , 2007, WWW '07.

[4]  Jeffrey F. Naughton,et al.  Set Containment Joins: The Good, The Bad and The Ugly , 2000, VLDB.

[5]  Guoliang Li,et al.  Can we beat the prefix filtering?: an adaptive framework for similarity join and search , 2012, SIGMOD Conference.

[6]  Timos K. Sellis,et al.  A combination of trie-trees and inverted files for the indexing of set-valued attributes , 2006, CIKM '06.

[7]  Xuemin Lin,et al.  TT-Join: Efficient Set Containment Join , 2017, 2017 IEEE 33rd International Conference on Data Engineering (ICDE).

[8]  Stefanie Rinderle-Ma,et al.  Dealing with change in process choreographies: Design and implementation of propagation algorithms☆ , 2015, Inf. Syst..

[9]  Vikram Pudi,et al.  Using Prefix-Trees for Efficiently Computing Set Joins , 2005, DASFAA.

[10]  Hector Garcia-Molina,et al.  Adaptive algorithms for set containment joins , 2003, TODS.

[11]  Hector Garcia-Molina,et al.  Index structures for selective dissemination of information under the Boolean model , 1994, TODS.

[12]  Nikos Mamoulis,et al.  Set containment join revisited , 2015, Knowledge and Information Systems.

[13]  Nikos Mamoulis,et al.  Efficient processing of joins on set-valued attributes , 2003, SIGMOD '03.

[14]  Jeffrey Xu Yu,et al.  Efficient similarity joins for near-duplicate detection , 2011, TODS.

[15]  Guoliang Li,et al.  An Efficient Partition Based Method for Exact Set Similarity Joins , 2015, Proc. VLDB Endow..

[16]  Vipin Kumar,et al.  Multi-view ensemble learning: an optimal feature set partitioning for high-dimensional data classification , 2015, Knowledge and Information Systems.

[17]  Ling Shao,et al.  LCJoin: Set Containment Join via List Crosscutting , 2019, 2019 IEEE 35th International Conference on Data Engineering (ICDE).

[18]  Nikolaus Augsten,et al.  An Empirical Evaluation of Set Similarity Join Techniques , 2016, Proc. VLDB Endow..

[19]  Yu Zhang,et al.  Selectivity Estimation by Batch-Query based Histogram and Parametric Method , 2007, ADC.

[20]  Jiaheng Lu,et al.  Efficient Merging and Filtering Algorithms for Approximate String Searches , 2008, 2008 IEEE 24th International Conference on Data Engineering.

[21]  Raghav Kaushik,et al.  Efficient exact set-similarity joins , 2006, VLDB.

[22]  Cédric du Mouza,et al.  Subscription indexes for web syndication systems , 2012, EDBT '12.

[23]  Parag Agrawal,et al.  On indexing error-tolerant set containment , 2010, SIGMOD Conference.

[24]  Sven Helmer,et al.  PIEJoin: Towards Parallel Set Containment Joins , 2016, SSDBM.

[25]  Sven Helmer,et al.  Evaluation of Main Memory Join Algorithms for Joins with Set Comparison Join Predicates , 1996, VLDB.

[26]  Jianzhong Li,et al.  FreshJoin: An Efficient and Adaptive Algorithm for Set Containment Join , 2019, APWeb/WAIM.

[27]  Hector Garcia-Molina,et al.  Divide-and-Conquer Algorithm for Computing Set Containment Joins , 2002, EDBT.

[28]  Renée J. Miller,et al.  LSH Ensemble: Internet-Scale Domain Search , 2016, Proc. VLDB Endow..

[29]  Jan Hidders,et al.  Efficient and scalable trie-based algorithms for computing set containment relations , 2015, 2015 IEEE 31st International Conference on Data Engineering.

[30]  Timos K. Sellis,et al.  Efficient answering of set containment queries for skewed item distributions , 2011, EDBT/ICDT '11.

[31]  Xuemin Lin,et al.  Top-k Set Similarity Joins , 2009, 2009 IEEE 25th International Conference on Data Engineering.