PEL-BERT: A Joint Model for Protocol Entity Linking

Pre-trained models such as BERT are widely used in NLP tasks and are fine-tuned to improve the performance of various NLP tasks consistently. Nevertheless, the fine-tuned BERT model trained on our protocol corpus still has a weak performance on the Entity Linking (EL) task. In this paper, we propose a model that joints a fine-tuned language model with an RFC Domain Model. Firstly, we design a Protocol Knowledge Base as the guideline for protocol EL. Secondly, we propose a novel model, PEL-BERT, to link named entities in protocols to categories in Protocol Knowledge Base. Finally, we conduct a comprehensive study on the performance of pre-trained language models on descriptive texts and abstract concepts. Experimental results demonstrate that our model achieves state-of-the-art performance in EL on our annotated dataset, outperforming all the baselines.

[1]  Cristina Nita-Rotaru,et al.  Leveraging Textual Specifications for Grammar-based Fuzzing of Network Protocols , 2019, AAAI.

[2]  Xiaolong Wang,et al.  Modeling Mention, Context and Entity with Neural Networks for Entity Disambiguation , 2015, IJCAI.

[3]  Geoffrey E. Hinton,et al.  Zero-shot Learning with Semantic Output Codes , 2009, NIPS.

[4]  Samuel Broscheit,et al.  Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking , 2019, CoNLL.

[5]  Yuval Marom,et al.  Experiments with Sentence Classification , 2006, ALTA.

[6]  Gerhard Weikum,et al.  Robust Disambiguation of Named Entities in Text , 2011, EMNLP.

[7]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2000, J. Mach. Learn. Res..

[8]  Jiuyang Tang,et al.  Entity Linking on Chinese Microblogs via Deep Neural Network , 2018, IEEE Access.

[9]  Eric Medvet,et al.  Active Learning of Regular Expressions for Entity Extraction , 2018, IEEE Transactions on Cybernetics.

[10]  Li Li,et al.  Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records , 2016, Scientific Reports.

[11]  Jaime G. Carbonell,et al.  A Little Annotation does a Lot of Good: A Study in Bootstrapping Low-resource Named Entity Recognizers , 2019, EMNLP.

[12]  Yang Feng,et al.  Bridging the Gap between Training and Inference for Neural Machine Translation , 2019, ACL.

[13]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[14]  Jimmy J. Lin,et al.  Rethinking Complex Neural Network Architectures for Document Classification , 2019, NAACL.

[15]  Dan Klein,et al.  Capturing Semantic Similarity for Entity Linking with Convolutional Neural Networks , 2016, NAACL.

[16]  Konrad P. Körding,et al.  Claim Extraction in Biomedical Publications using Deep Discourse Model and Transfer Learning , 2019, ArXiv.

[17]  Yoon Kim,et al.  Convolutional Neural Networks for Sentence Classification , 2014, EMNLP.

[18]  Luke S. Zettlemoyer,et al.  Deep Contextualized Word Representations , 2018, NAACL.

[19]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[20]  Jimmy J. Lin,et al.  DocBERT: BERT for Document Classification , 2019, ArXiv.

[21]  Oscar Mayora-Ibarra,et al.  Stress Modelling Using Transfer Learning in Presence of Scarce Data , 2015, AmIHEALTH.

[22]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[23]  Xianpei Han,et al.  Learning to Bootstrap for Entity Set Expansion , 2019, EMNLP.

[24]  Thomas Hofmann,et al.  End-to-End Neural Entity Linking , 2018, CoNLL.

[25]  Xuanjing Huang,et al.  How to Fine-Tune BERT for Text Classification? , 2019, CCL.

[26]  Zita Marinho,et al.  Joint Learning of Named Entity Recognition and Entity Linking , 2019, ACL.

[27]  Mahak Motwani,et al.  Text Categorization Comparison between Simple BPNN and Combinatorial Method of LSI and BPNN , 2014 .

[28]  Jiuyang Tang,et al.  Collective List-Only Entity Linking: A Graph-Based Approach , 2018, IEEE Access.

[29]  Iryna Gurevych,et al.  Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , 2018, ACL 2018.