Can Annihilating dark matter be lighter than a few GeVs