Validating Label Consistency in NER Data Annotation

Data annotation plays a crucial role in ensuring your named entity recognition (NER) projects are trained with the right information to learn from. Producing the most accurate labels is a challenge due to the complexity involved with annotation. Label inconsistency between multiple subsets of data annotation (e.g., training set and test set, or multiple training subsets) is an indicator of label mistakes. In this work, we present an empirical method to explore the relationship between label (in-)consistency and NER model performance. It can be used to validate the label consistency (or catches the inconsistency) in multiple sets of NER data annotation. In experiments, our method identified the label inconsistency of test data in SCIERC and CoNLL03 datasets (with 26.7% and 5.4% label mistakes). It validated the consistency in the corrected version of both datasets.

[1]  Qingkai Zeng,et al.  Enhancing Taxonomy Completion with Concept Generation via Fusing Relational Representations , 2021, KDD.

[2]  Tim Weninger,et al.  Tri-Train: Automatic Pre-Fine Tuning between Pre-Training and Fine-Tuning for SciNER , 2020, FINDINGS.

[3]  Wenhao Yu,et al.  Identifying Referential Intention with Heterogeneous Contexts , 2020, WWW.

[4]  Sle Sle,et al.  The General , 2020, Making Patton.

[5]  Qingkai Zeng,et al.  Biomedical Knowledge Graphs Construction From Conditional Statements , 2020, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[6]  Wenhao Yu,et al.  Faceted Hierarchy: A New Graph Type to Organize Scientific Concepts and a Construction Method , 2019, EMNLP.

[7]  Zihan Wang,et al.  CrossWeigh: Training Named Entity Tagger from Imperfect Annotations , 2019, EMNLP.

[8]  Mari Ostendorf,et al.  A general framework for information extraction using dynamic span graphs , 2019, NAACL.

[9]  Mari Ostendorf,et al.  Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction , 2018, EMNLP.

[10]  Xiang Ren,et al.  Empower Sequence Labeling with Task-Aware Neural Language Model , 2017, AAAI.

[11]  Chris Dyer,et al.  Neural Architectures for Named Entity Recognition , 2016, NAACL.

[12]  Eduard H. Hovy,et al.  End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF , 2016, ACL.

[13]  Christopher D. Manning.  Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? , 2011, CICLing.

[14]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[15]  Karel Oliva,et al.  (Semi-)Automatic Detection of Errors in PoS-Tagged Corpora , 2002, COLING.

[16]  Eleazar Eskin,et al.  Detecting Errors within a Corpus using Anomaly Detection , 2000, ANLP.