Holistic Data Profiling: Simultaneous Discovery of Various Metadata

Data proling is the discipline of examining an unknown dataset for its structure and statistical information. It is a preprocessing step in a wide range of applications, such as data integration, data cleansing, or query optimization. For this reason, many algorithms have been proposed for the discovery of dierent kinds of metadata. When analyzing a dataset, these proling algorithms are often applied in sequence, but they do not support one another, for instance, by sharing I/O cost or pruning information. We present the holistic algorithm Muds, which jointly discovers the three most important metadata: inclusion dependencies, unique column combinations, and functional dependencies. By sharing I/O cost and data structures across the dierent discovery tasks, Muds can clearly increase the eciency of traditional sequential data proling. The algorithm also introduces novel inter-task pruning rules that build upon dierent types of metadata, e.g., unique column combinations to infer functional dependencies. We evaluate Muds in detail and compare it against the sequential execution of state-of-the-art algorithms. A comprehensive evaluation shows that our holistic algorithm outperforms the baseline by up to factor 48 on datasets with favorable pruning conditions.