Intrinsic and environmental effects on the interference properties of a high-performance quantum dot single-photon source