Evolution of Intermediate-Mass Black Hole X-Ray Binaries