Suppression of superconductivity due to non-perturbative saddle points in the nonlinear $\sigma$-model