Effective-Hamiltonian Theory of Open Quantum Systems at Strong Coupling