The Legacy for Herschel from ISO and Spitzer