Modeling of diffraction of electromagnetic waves on periodic inhomogeneities by a finite element method coupled with the Rayleigh expansion