Semiclassical origin of the spectral gap for transfer operators of a partially expanding map