Growth of stellar mass black holes in galactic nuclei