Variational approach for extraction of eigenmodes in layered waveguides