The Token Distribution Filter for Approximate String Membership

A common application over web data is to find all the strings in a collection of pages that match strings in a given dictionar y. We consider the problem of extracting all the strings or substr ings in a document (or a page) that approximately match some string in a given dictionary. The current state-of-art approach for th is problem involves first applying an approximate, fast filter, then app lying a more expensive exact verification algorithm to the strings t hat survive the filter. Many string filters, such as the length filter a nd prefix filter, have been proposed. However, we find many string filter s are ineffective or inefficient in some problem scenarios. In thi s paper, we propose a new filter, the TDF (token distribution filter). W e conduct experiments on both synthetic and real data sets, and show that for a wide class of problems it performs better than previously proposed filters.