Stability of relativistic tidal response against small potential modification