Dynamical Friction on Galactic Center Star Clusters with an Intermediate-Mass Black Hole