Exact solutions of a multi-component anyon model with SU(N) invariance