Fundamental limitations to high-precision tests of the universality of free fall by dropping atoms