Ensemble-based gradient inference for particle methods in optimization and sampling