Ruling out minimal anomaly free $\mathrm{U}(1)$ extensions of the Standard Model