Smooth Feshbach map and operator-theoretic renormalization group methods