Efficient Adapters for Giant Speech Models