Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning