Quenched equals annealed at leading order in the colored SYK model