Query processing in a DHT-based universal storage: the world as a peer-to-peer database

With the growth of the Internet, novel applications scale significantly in both, data and the number of users. In parallel, the size and local expansion of underlying network infrastructures increase as well. A wide range of such applications provides access to huge data collections to a likewise huge number of participants. If these data are of public interest and open to the public, according applications are classified as Public Data Management. Often, data are also open to extension and modification. Thus, data get into existence through collaboration and are maintained by the community. Structured data and according access methods are often a mandatory requirement to make these systems work. In doing so, a high degree of heterogeneity is very probable. Features known from traditional database and data integration systems are often wanted, expected, or even required to handle such data collections meaningfully. The applicability of centralized approaches is limited in scale and cannot keep with the ever-growing requirements. Distribution of data and load is an intuitive and mandatory consequence. Distribution in small scale, such as in clusters and locally encapsulated systems, gets to its limits as well. The “kill it with iron” strategy is not applicable, because often no instance or single user is willed and enabled to bear the immense costs of resulting systems. In turn, Public Data Management applications also result in lowered requirements, such as relaxed guarantees in consistency and availability. This turns the idea of a total distribution of all data and load into a promising and feasible approach. Even if existing systems already provide according features, these are in general by far simplified and strongly limited. A wide range of requirements of database-like distributed systems is still neither supported nor enabled. This work focuses on the requirements and unsolved challenges that occur specifically in the context of query processing. Besides the pure efficiency of query processing, this also includes issues like query expressiveness and practicability of certain concepts. We first identify these and other challenges and motivate the contributions of this work. Afterwards, we propose according techniques and approaches for efficient query processing in widely distributed systems. We suggest a layered architecture that makes use of the advantages of decentralized Peer-to-Peer approaches. Based on this, a vertical triple model, and an according query algebra, we propose a physical query algebra. This algebra comprises operators that make excessive use of parallelism and other paradigms of distributed query processing. Further, we present techniques for processing complex query plans that represent user queries. The query framework is extended by an appropriate cost model to enable meaningful query planning. We also discuss guarantees and predictability. In this context, we introduce a light-weight and flexible approach for estimating the completeness of partly query answers. An extensive evaluation based on a reference implementation shows the applicability and correctness of the proposed concepts. This highlights the valuable contribution for the development of large-scale distributed data systems. Marcel Karnstedt: Query Processing in a DHT-Based Universal Storage vii