Rademacher Complexity for Adversarially Robust Generalization Supplementary Material

A recent line of work analyzes the convergence and generalization problems in distributional robust optimization (DRO) [5, 14, 19]. The notion of DRO differs from ours, since DRO considers the setting where the distribution of the input data is being perturbed, while we consider the perturbation in the feature space. Farnia et al. [6] study the generalization problem when the attack algorithm of the adversary is provided to the learner, which is also a weaker notion than our problem. A few other lines of work have been trying to conduct theoretical analysis of adversarial examples. Wang et al. [21] analyze the adversarial robustness of nearest neighbors estimator. Papernot et al. [17] try to demonstrate the unavoidable trade-offs between accuracy in the natural setting and the resilience to adversarial attacks, and this trade-off is further studied by Tsipras et al. [20] through some constructive examples of distributions. Fawzi et al. [7] analyze adversarial robustness of fixed classifiers, in contrast to our generalization analysis. Fawzi et al. [8] construct examples of distributions with large latent variable space such that adversarially robust classifiers do not exist; here we argue that these examples may not explain the fact that adversarially perturbed images can usually be recognized by humans. Bubeck et al. [3] try to explain the hardness of learning an adversarially robust model from the computational constraints under the statistical query model. Another recent line of work explains the existence of adversarial examples via high dimensional geometry and concentration of measure [9, 4, 15]. These works provide examples where adversarial examples provably exist as long as the test error of a classifier is non-zero. Our results show that adding `1 constraints on the weights of neural networks can improve the generalization gap in the adversarial setting. This is consistent with some recent works which show that sparsified neural networks may improve adversarial robustness [11, 10]. In earlier work, Bagnell proposed a concept of robust supervised learning [1]; robust optimization has been studied in Lasso [23] and SVM [24] problems. Xu and Mannor [22] make the connection between algorithmic robustness and generalization property in the natural setting, whereas our work focus on generalization in the adversarial setting.