Formation of dust vortices in a nuclear-excited plasma