When to Trust Aggregated Gradients: Addressing Negative Client Sampling in Federated Learning