Direct determination of effective Hamiltonians by wave-operator methods. I. General formalism