Geometry of $\nu $-Tamari lattices in types $A$ and $B$