The infrared compactness-temperature relation for quiescent and starburst galaxies