The IR stability of de Sitter QFT: physical initial conditions