BAGEL: Bootstrapping Agents by Guiding Exploration with Language