Preserving Modality Structure Improves Multi-Modal Learning