Universality for first passage percolation on sparse uniform and rank-1 random graphs