Potter''s Wheel: An Interactive Framework for Data Transformation and Cleaning

An important step in data warehousing and Enterprise Data Integration is cleaning data of discrepancies in structure and content. Current commercial solutions for data cleaning involve many iterations of time-consuming “auditing” to find errors, and long-running transformations to fix them. Users need to endure long waits and often write complex transformation programs. In this paper, we present an interactive data cleaning system that tightly integrates transformation and discrepancy detection. Users gradually build transformations by adding or undoing transforms, in a intuitive, graphical manner through a spreadsheet-like interface; the effect of a transform is shown at once on records visible on screen. In the background, the system automatically infers the structure of the data in terms of user-defined domains and applies suitable algorithms to check the data for discrepancies, flagging them as they are found. This allows users to gradually construct a transformation as discrepancies are found, and clean the data without writing complex programs or enduring long delays. We choose and adapt a small set of transforms from the literature and describe methods for their graphical specification and interactive application. We combine the Minimum Description Length principle with the traditional database notion of user-defined types to automatically extract suitable structures for data values, in an extensible fashion. Such structure extraction is also applied in the graphical specification of transforms, to infer transforms from examples. We also describe methods for optimizing the final sequence of transforms to minimize memory allocations