Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models