Less can be more in contrastive learning