Deformations of the circular Wilson loop and spectral (in)dependence