Mixing properties of colorings of the $\mathbb{Z}^d$ lattice