Coherent radiation of relativistic nonlinear Thomson scattering