One Neuron to Fool Them All