Modeling radiation with an efficient hybrid finite-element integral-equation waveguide mode-matching technique