Niching and evolutionary transitions in multi-agent systems