Revisiting the Fragility of Influence Functions