Overcoming the Pitfalls of Prediction Error in Operator Learning for Bilevel Planning