Instability of a rotating shear layer in the transonic regime