Instability of strongly nonlinear waves in vortex flows