Isometry groups of simply connected manifolds of nonpositive curvature II