Processing recursive XQuery over XML streams: The Raindrop approach

XML stream applications bring the challenge of efficiently processing queries on sequentially accessible token-based data. For efficient processing of queries, we need to ensure that memory usage stays low. This in turn requires that we avoid holding data in the query buffer, by outputting it at the earliest possible time. In this paper, we propose a new class of stream algebra operators for efficient recursive XQuery stream processing. Our plan generator will analyze the query, and the schema when available to determine which join operators in the query need recursive join support and thus can plug in the more inexpensive just-in-time structural join whenever possible. In particular, we propose two strategies for implementing structural joins: (a) the just-in-time structural join strategy efficiently processes joins over non-recursive XML token streams; and (b) the recursive structural join strategy supports structural joins over recursive XML substreams, however, at an added cost of generating and comparing tuple-level ID. Both structural join strategies are complemented by an automata-driven invocation mechanism that triggers the execution of each join process at the first possible moment upon recognizing the end of the targeted input stream subelement. Further, we design this StructuralJoin operator itself to be context-aware. The operator is capable of at run-time switching from the efficient just-in-time join strategy for elements that are recognized to be non-recursive to the more powerful ID-based structural join strategy for elements that are identified to be recursive. We incorporate the proposed techniques into the Raindrop stream engine. We also report on experimental studies we conducted using the ToXgene benchmark that demonstrate that the performance improvements of the techniques.