Extreme and Topological Nonlinear Optics of Open Systems