Relaxation dynamics and dissipative phase transition in quantum oscillators with period tripling