Stability of the perturbative vacuum against spatial variations of the Polyakov loop