Rescuing single field inflation from the swampland