Nonlinear hybrid vector finite element method for modeling wave propagation in nonlinear media