Space and Time Scalability of Duplicate Detection in Graph Data

Duplicate detection consists in determining differentrepresentations of real-world objects in a database. Recent research has considered the use of relatio nships among object representations to improve duplicate detection. In the general case where re lationships form a graph, research has mainly focused on duplicate detection quality/effecti v ness. Scalability has been neglected so far, even though it is crucial for large real-world duplic ate detection tasks. In this paper we scale up duplicate detection in graph data (D DG) to large amounts of data and pairwise comparisons, using the support of a relational database system. To this end, we first generalize the process of DDG. We then present how to sca le algorithms for DDG in space (amount of data processed with limited main memory) and in ti me. Finally, we explore how complex similarity computation can be performed efficientl y. Experiments on data an order of magnitude larger than data considered so far in DDG clearly s how that our methods scale to large amounts of data not residing in main memory.