Speeding up Inference with User Simulators through Policy Modulation