Theory of three-magnon interaction in a vortex-state magnetic nanodot