Young star clusters immersed in intermediate-age fields in the Small Magellanic Cloud