General limit to thermodynamic annealing performance.