On vanishing class sizes in finite groups