Decoherence times of universal two-qubit gates due to structured noise