Innovative Windows for Duplicate Detection

Duplicate detection is the special case of data matching that discovers groups of records within a single database that belong to same real world entity. It is also an inevitable part of data cleansing because duplicate records can strongly influence the results of later data mining or processing. In this process, one record is compared to all other records. Different data representations, formats, terminologies and data entry errors make this task complex. Involvement of heavy volume databases adds more complexity. To reduce comparison of records, indexing algorithms are used that partition the data and perform comparisons with in that partition. Sorted Neighborhood Method (SNM) is a standard indexing algorithm that sorts dataset by using defined “sorting key” and moves fixed size window to compare records within that window. Duplicate Count Strategy-Multi record increase (DCS++) is latest improvement in SNM that adapts the window size for every duplicate in current window. We propose Innovative Windows (Inn Win) algorithm that assumes i) detected duplicate in sorted dataset raises the probability of finding more duplicates in neighborhood ii) Series of consecutive non-duplicates drops the probability of duplicates in neighborhood. Using this concept, it adapts window both for duplicates and non-duplicates and avoids unnecessary comparisons without losing effectiveness. We prove that Inn Win is a better alternative in windowing algorithms.