ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data