Scattering from a Rayleigh layer with an irregular interface