Robustness and diffusion of pointer states