Hilbert Space Diffusion in Systems with Approximate Symmetries