Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP