Limits to two-spin-qubit gate fidelity from thermal and vacuum fluctuations