Addressing Negative Transfer in Diffusion Models