Quantum phase transition in the U(3) boson system