Vortex phases of an unconventional superconductor with two components order parameter