Nonconvex separation theorems and some applications in vector optimization