A Practioner's Guide to Evaluating Entity Resolution Results

Entity resolution (ER) is the task of identifying records belonging to the same entity (e.g. individual, group) across one or multiple databases. Ironically, it has multiple names: deduplication and record linkage, among others. In this paper we survey metrics used to evaluate ER results in order to iteratively improve performance and guarantee sufficient quality prior to deployment. Some of these metrics are borrowed from multi-class classification and clustering domains, though some key differences exist differentiating entity resolution from general clustering. Menestrina et al. empirically showed rankings from these metrics often conflict with each other, thus our primary motivation for studying them. This paper provides practitioners the basic knowledge to begin evaluating their entity resolution results.

[1]  Andrew McCallum,et al.  An Integrated, Conditional Model of Information Extraction and Coreference with Appli , 2004, UAI.

[2]  Breck Baldwin,et al.  Algorithms for Scoring Coreference Chains , 1998 .

[3]  Sofus A. Macskassy,et al.  Record Linkage Measures in an Entity Centric World , 2009 .

[4]  L. Getoor,et al.  Entity Resolution Evaluation Measures , 2012 .

[5]  Julia Hirschberg,et al.  V-Measure: A Conditional Entropy-Based External Cluster Evaluation Measure , 2007, EMNLP.

[6]  H. Bourlard,et al.  Improved Unknown-Multiple Speaker clustering using HMM , 2002 .

[7]  W. Marsden.  I and J , 2012 .

[8]  Marina Meila,et al.  Comparing Clusterings by the Variation of Information , 2003, COLT.

[9]  W. Winkler.  Overview of Record Linkage and Current Research Directions , 2006 .

[10]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[11]  Peter Christen,et al.  Quality and Complexity Measures for Data Linkage and Deduplication , 2007, Quality Measures in Data Mining.

[12]  Jennifer Widom,et al.  Swoosh: a generic approach to entity resolution , 2008, The VLDB Journal.

[13]  C. Lee Giles,et al.  Efficient Name Disambiguation for Large-Scale Databases , 2006, PKDD.

[14]  Hector Garcia-Molina,et al.  Evaluating entity resolution results , 2010, Proc. VLDB Endow..

[15]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[16]  Lynette Hirschman,et al.  A Model-Theoretic Coreference Scoring Scheme , 1995, MUC.

[17]  Herbert Gish,et al.  Clustering speakers by their voices , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[18]  M. Hosszú,et al.  On the functional equation F(x+y,z)+F(x,y)=F(x,y+z)+F(y,z) , 1971 .

[19]  Xiaoqiang Luo,et al.  On Coreference Resolution Performance Metrics , 2005, HLT.

[20]  Ashwin Machanavajjhala,et al.  Entity Resolution: Theory, Practice & Open Challenges , 2012, Proc. VLDB Endow..