Trading-off Information Modalities in Zero-shot Classification