Quantum phase transition in an atom–molecule conversion system with atomic transition