Mitigating Bias in Visual Transformers via Targeted Alignment