Improved Estimation of Concentration Under $\ell_p$-Norm Distance Metrics Using Half Spaces

Concentration of measure has been argued to be the fundamental cause of adversarial vulnerability. Mahloujifar et al. (2019b) presented an empirical way to measure the concentration of a data distribution using samples, and employed it to find lower bounds on intrinsic robustness for several benchmark datasets. However, it remains unclear whether these lower bounds are tight enough to provide a useful approximation for the intrinsic robustness of a dataset. To gain a deeper understanding of the concentration of measure phenomenon, we first extend the Gaussian Isoperimetric Inequality to non-spherical Gaussian measures and arbitrary `p-norms (p ≥ 2). We leverage these theoretical insights to design a method that uses half-spaces to estimate the concentration of any empirical dataset under `p-norm distance metrics. Our proposed algorithm is more efficient than Mahloujifar et al. (2019b)’s, and our experiments on synthetic datasets and image benchmarks demonstrate that it is able to find much tighter intrinsic robustness bounds. These tighter estimates provide further evidence that rules out intrinsic dataset concentration as a possible explanation for the adversarial vulnerability of state-of-the-art classifiers.

[1]  Elvis Dohmatob,et al.  Generalized No Free Lunch Theorem for Adversarial Robustness , 2018, ICML.

[2]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[3]  Sven Gowal,et al.  Scalable Verified Training for Provably Robust Image Classification , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  M. Talagrand.  Concentration of measure and isoperimetric inequalities in product spaces , 1994, math/9406212.

[5]  Saeed Mahloujifar,et al.  Empirically Measuring Concentration: Fundamental Limits on Intrinsic Robustness , 2019, NeurIPS.

[6]  V. Sudakov,et al.  Extremal properties of half-spaces for spherically invariant measures , 1978 .

[7]  Hamza Fawzi,et al.  Adversarial vulnerability for any classifier , 2018, NeurIPS.

[8]  Saeed Mahloujifar,et al.  The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measure , 2018, AAAI.

[9]  Saeed Mahloujifar,et al.  Adversarial Risk and Robustness: General Definitions and Implications for the Uniform Distribution , 2018, NeurIPS.

[10]  Michael I. Jordan,et al.  Theoretically Principled Trade-off between Robustness and Accuracy , 2019, ICML.

[11]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[12]  J. Zico Kolter,et al.  Certified Adversarial Robustness via Randomized Smoothing , 2019, ICML.

[13]  Jinghui Chen,et al.  Understanding the Intrinsic Robustness of Image Distributions using Conditional Generative Models , 2020, AISTATS.

[14]  C. Borell.  The Brunn-Minkowski inequality in Gauss space , 1975 .

[15]  J. Zico Kolter,et al.  Provable defenses against adversarial examples via the convex outer adversarial polytope , 2017, ICML.

[16]  Tom Goldstein,et al.  Are adversarial examples inevitable? , 2018, ICLR.

[17]  M. Raïssouli,et al.  Various Proofs for the Decrease Monotonicity of the Schatten's Power Norm, Various Families of R n Norms and Some Open Problems , 2010 .

[18]  Ananthram Swami,et al.  Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks , 2015, 2016 IEEE Symposium on Security and Privacy (SP).

[19]  M. Ledoux,et al.  Isoperimetry and Gaussian analysis , 1996 .

[20]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[21]  Ilya P. Razenshteyn,et al.  Adversarial examples from computational constraints , 2018, ICML.

[22]  Daniel Cullina,et al.  Lower Bounds on Adversarial Robustness from Optimal Transport , 2019, NeurIPS.

[23]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.