Interplay between sign problem and $Z_3$ symmetry in three-dimensional Potts models