Fidelity of Fock-state-encoded qubits subjected to continuous-variable Gaussian processes