Optimizing over a set of manifolds