Finite temperature effects in Coulomb blockade quantum dots and signatures of spectral scrambling