General theory for the spectral evolution of nonlinear Rayleigh waves