All linear operators leaving the unitary group invariant