Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval