Decoding Structure-Spectrum Relationships with Physically Organized Latent Spaces