Observation of a non-Hermitian phase transition in an optical quantum gas