On the Optimality of Velocity Averaging Lemmas