Why is the Milky Way X-factor constant?