Degradation of light carrying orbital angular momentum by ballistic scattering