Limits to Quantum Gate Fidelity from Near-Field Thermal and Vacuum Fluctuations