Dynamical and collisional constraints on a stochastic late veneer on the terrestrial planets