How Do Your Biomedical Named Entity Models Generalize to Novel Entities?

The number of biomedical literature on new biomedical concepts is rapidly increasing, which necessitates a reliable biomedical named entity recognition (BioNER) model for identifying new and unseen entity mentions. However, it is questionable whether existing BioNER models can effectively handle them. In this work, we systematically analyze the three types of recognition abilities of BioNER models: memorization, synonym generalization, and concept generalization. We find that although BioNER models achieve state-of-the-art performance on BioNER benchmarks based on overall performance, they have limitations in identifying synonyms and new biomedical concepts such as COVID-19. From this observation, we conclude that existing BioNER models are overestimated in terms of their generalization abilities. Also, we identify several difficulties in recognizing unseen mentions in BioNER and make the following conclusions: (1) BioNER models tend to exploit dataset biases, which hinders the models’ abilities to generalize, and (2) several biomedical names have novel morphological patterns with little name regularity such as COVID-19, and models fail to recognize them. We apply a current statistics-based debiasing method to our problem as a simple remedy and show the improvement in generalization to unseen mentions. We hope that our analyses and findings would be able to facilitate further research into the generalization capabilities of NER models in a domain where their reliability is of utmost importance.