Eigenmode calculation of dielectric waveguides near cutoff using the Rayleigh-Ritz method with rational basis functions