The Signature of Solar Activity in the Infrared Spectral Irradiance