Wildly perturbed manifolds: norm resolvent and spectral convergence