Fisher information lower bounds for sampling