Self-consistent radiative effect on relativistic electromagnetic particle acceleration