Parallelizing windowed stream joins in a shared-nothing cluster

The availability of a large number of processing nodes in a parallel and distributed computing environment enables sophisticated real time processing over high speed data streams, as required by many emerging applications. Sliding window stream joins are among the most important operators in a stream processing system. In this paper, we consider the issue of parallelizing a sliding window stream join operator over a shared nothing cluster. We propose a Bulk Synchronous Processing (BSP) framework, based on a fixed or predefined sequence of communication, to distribute the join processing loads over a shared-nothing cluster. We consider various processing and communication overheads while scaling over a large number of nodes, and propose solution methodologies to cope with the issues.We implement the algorithm over a cluster using a message passing system, and present the experimental results showing the effectiveness of the join processing algorithm.

[1]  Michael Stonebraker,et al.  Contract-Based Load Management in Federated Distributed Systems , 2004, NSDI.

[2]  Christos Faloutsos,et al.  Capturing the spatio-temporal behavior of real traffic data , 2002, Perform. Evaluation.

[3]  WidomJennifer,et al.  Continuous queries over data streams , 2001 .

[4]  Jennifer Widom,et al.  Adaptive caching for continuous queries , 2005, 21st International Conference on Data Engineering (ICDE'05).

[5]  Tore Risch,et al.  Customizable Parallel Execution of Scientific Stream Queries , 2005, VLDB.

[6]  Patrick Valduriez,et al.  StreamCloud: A Large Scale Data Streaming System , 2010, 2010 IEEE 30th International Conference on Distributed Computing Systems.

[7]  Michael Stonebraker,et al.  The Case for Shared Nothing , 1985, HPTS.

[8]  Masaru Kitsuregawa,et al.  Bucket Spreading Parallel Hash: A New, Robust, Parallel Hash Join Method for Data Skew in the Super Database Computer (SDC) , 1990, VLDB.

[9]  Philip S. Yu,et al.  A Parallel Hash Join Algorithm for Managing Data Skew , 1993, IEEE Trans. Parallel Distributed Syst..

[10]  Mostafa Bamha,et al.  Frequency-Adaptive Join for Shared Nothing Machines , 1999, Scalable Comput. Pract. Exp..

[11]  David J. DeWitt,et al.  Practical Skew Handling in Parallel Joins , 1992, VLDB.

[12]  Goetz Graefe,et al.  Encapsulation of parallelism in the Volcano query processing system , 1990, SIGMOD '90.

[13]  Michael Stonebraker,et al.  Load Shedding in a Data Stream Manager , 2003, VLDB.

[14]  Joseph M. Hellerstein,et al.  Flux: an adaptive partitioning operator for continuous query systems , 2003, Proceedings 19th International Conference on Data Engineering (Cat. No.03CH37405).

[15]  Michael Stonebraker,et al.  Monitoring Streams - A New Class of Data Management Applications , 2002, VLDB.

[16]  A GibsonGarth,et al.  Network attached storage architecture , 2000 .

[17]  J DeWittDavid,et al.  A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment , 1989 .

[18]  Ronald Fagin,et al.  Extendible hashing—a fast access method for dynamic files , 1979, ACM Trans. Database Syst..

[19]  Jennifer Widom,et al.  Continuous queries over data streams , 2001, SGMD.

[20]  Philip S. Yu,et al.  ViCo: an adaptive distributed video correlation system , 2006, MM '06.

[21]  GraefeGoetz.  Encapsulation of parallelism in the Volcano query processing system , 1990 .

[22]  Philip S. Yu,et al.  Adaptive Load Diffusion for Multiway Windowed Stream Joins , 2007, 2007 IEEE 23rd International Conference on Data Engineering.

[23]  Erhard Rahm,et al.  Dynamic Multi-Resource Load Balancing in Parallel Database Systems , 1995, VLDB.

[24]  Liang Chen,et al.  Handling data skew in parallel joins in shared-nothing systems , 2008, SIGMOD Conference.

[25]  Ying Xing,et al.  Dynamic load distribution in the Borealis stream processor , 2005, 21st International Conference on Data Engineering (ICDE'05).

[26]  W. Richard Stevens,et al.  TCP/IP Illustrated, Volume 1: The Protocols , 1994 .

[27]  Greg Burns,et al.  LAM: An Open Cluster Environment for MPI , 2002 .

[28]  Arthur M. Keller,et al.  Adaptive parallel hash join in main-memory databases , 1991, [1991] Proceedings of the First International Conference on Parallel and Distributed Information Systems.

[29]  Xiaofang Zhou,et al.  Handling data skew in parallel hash join computation using two-phase scheduling , 1995, Proceedings 1st International Conference on Algorithms and Architectures for Parallel Processing.

[30]  Kien A. Hua,et al.  Handling Data Skew in Multiprocessor Database Computers Using Partition Tuning , 1991, VLDB.

[31]  FaloutsosChristos,et al.  Capturing the spatio-temporal behavior of real traffic data , 2002 .

[32]  Philip S. Yu,et al.  Adaptive Load Diffusion for Stream Joins , 2005, Middleware.

[33]  Tim Kraska,et al.  Stormy: an elastic and highly available streaming service in the cloud , 2012, EDBT-ICDT '12.

[34]  Masaru Kitsuregawa,et al.  Dynamic Join Product Skew Handling for Hash-Joins in Shared-Nothing Database Systems , 1995, DASFAA.

[35]  Geoffrey C. Fox,et al.  MPIJAVA: An Object-Oriented JAVA Interface to MPI , 1999, IPPS/SPDP Workshops.

[36]  Philip S. Yu,et al.  Adaptive load shedding for windowed stream joins , 2005, CIKM '05.

[37]  Kenji Imasaki,et al.  An adaptive hash join algorithm on a network of workstations , 2002, Proceedings 16th International Parallel and Distributed Processing Symposium.

[38]  Kenneth Baclawski,et al.  Quickly generating billion-record synthetic databases , 1994, SIGMOD '94.

[39]  Jennifer Widom,et al.  Memory-Limited Execution of Windowed Stream Joins , 2004, VLDB.

[40]  Minos N. Garofalakis,et al.  Multi-dimensional resource scheduling for parallel queries , 1996, SIGMOD '96.

[41]  Rodney Van Meter,et al.  Network attached storage architecture , 2000, CACM.

[42]  Wei Han,et al.  Parallel Querying with Non-Dedicated Computers , 2005, VLDB.