Variational boundary perturbation theory for enclosed quantum systems