Parallel computing techniques for high-performance probabilistic record linkage

Record linkage techniques are used to link together records from one or more data sets relating to the same entity, e.g. patient or customer. As data is often not primarily collected for data analysis purposes, a common unique identifi er is missing in many cases, and probabilistic linkage techniques have to be applied. Historical collections of administrative and other (health) data nowadays contain tens of millions of records, with new data being added at the rate of millions of records per year. Although improvements in available computing power have to some extent mitigated against the effects of this accelerating growth in the size of the data sets to be linked, large-scale probabilistic record linkage is still a slow and resource-intensive process. The ANU Data Mining Group is currently working in collaboration with Epidemiology and Surveillance Branch of the NSW Health Department on the development of improved techniques for probabilistic record linkage. Our main focus is the development of techniques that make good use of modern high-performance parallel computers, and the exploration of data mining and machine learning techniques to reduce the time consuming and tedious manual clerical review process for possible links. The developed software will be published under an open source software license. We hope to have prototype software available early in the second half of 2002.