Uniqueness of equivariant harmonic maps to symmetric spaces and buildings