The clustering of X-ray AGN at 0.5 < z < 4.5: host galaxies dictate dark matter halo mass