Phase transitions in frustrated XY model on a square lattice