Knowledge Distillation Techniques for Biomedical Named Entity Recognition

The limited amount of annotated biomedical literature and its peculiar characteristics make biomedical named entity recognition more challenging than standard named entity recognition. The multitask learning approach overcomes these limitations by training different related tasks simultaneously. It learns common features among different tasks by sharing some layers of the neural network architecture. For this reason, the multi-task model attains more generalization properties than a single task learning. The generalization of the multi-task model can be utilized to enhance other models’ results. In particular, knowledge distillation techniques make this possible in which one model supervises, through its learned generalization, another model during the training. This research analyzes the knowledge distillation approach and shows that a simple deep learning model performance can be leveraged through distilling the multi-task model’s generalization. Results show that our approach outperformed compared with the multi-task model and single task model. This demonstrates that our model learns more diverse features using the knowledge distillation approach. We also found our approach statistically better than multi-task model and single task model.