Semigroups with few endomorphisms