C$^*$-simple groups: amalgamated free products, HNN extensions, and fundamental groups of 3-manifolds