Scattering invariance for arbitrary polarizations protected by joint spatial-duality symmetries