Growth of Intermediate-Mass Black Holes in Globular Clusters