Finite element implementation of Bayliss-Turkel boundary operators in the three-dimensional vector wave equation