Convection in a rotating layer with nearly insulating boundaries