Dynamical noise sensitivity for the voter model