Exact one-particle density matrix for SU($N$) fermionic matter-waves in the strong repulsive limit