Incremental entity fusion from linked documents

In many government applications, especially for intelligence and law-enforcement, we often find that information about entities, such as persons or even companies, are available in disparate data sources. For example, information distributed across passports, driving licences, bank accounts, and income tax documents that need to be resolved and fused to reveal a consolidated profile of an individual. In this paper we describe an algorithm to fuse documents that are highly likely to belong to the same entity by exploiting inter-document references in addition to attribute similarity. Our technique uses a combination of iterative graph-traversal, locality-sensitive hashing, iterative match-merge, and graph-clustering to discover unique entities based on a document corpus. Further, new sets of documents can be added incrementally while having to re-process only a small subset of a previously fused entity-document collection. We present performance and quality results via both Bayesian likelihood fusion as well as using Support Vector Machines to demonstrate benefit of using inter-document references, both to improve accuracy as well as for detecting attempts at deliberate obfuscation.