Early growth of high redshift black holes