Task-to-Task Transfer Learning with Parameter-Efficient Adapter

Many existing pre-trained language models have yielded strong performance on many NLP tasks. They depend on enough labeled data of downstream tasks, which are difficult to be trained on tasks with limited data. Transfer learning from large labeled task to narrow task based on the pre-trained language models can solve this problem. However, it always suffers from catastrophic forgetting. In this paper, we propose an effective task-to-task transfer learning method with parameter-efficient adapter based on pre-trained language model, which can be trained on new tasks without hindering the performance of those already learned. Our experiments include transfer learning from MNLI or SQUAD (as the source task) to some related small data tasks based on Bert. Experimental results show large gains in effectiveness over previous approaches on transfer learning and domain adaptation without forgetting. By adding less than 2.1% of the parameters, our method matches or outperforms vanilla fine-tuning and can overcome catastrophic forgetting.