Structure of the Space of Ground States in Systems with Non-Amenable Symmetries