Homotopy invariance of higher signatures and $3$-manifold groups