Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification

Adversarial robustness has become a fundamental requirement in modern machine learning applications. Yet, there has been surprisingly little statistical understanding so far. In this paper, we provide the first result of the optimal minimax guarantees for the excess risk for adversarially robust classification, under Gaussian mixture model proposed by \cite{schmidt2018adversarially}. The results are stated in terms of the Adversarial Signal-to-Noise Ratio (AdvSNR), which generalizes a similar notion for standard linear classification to the adversarial setting. For the Gaussian mixtures with AdvSNR value of $r$, we establish an excess risk lower bound of order $\Theta(e^{-(\frac{1}{8}+o(1)) r^2} \frac{d}{n})$ and design a computationally efficient estimator that achieves this optimal rate. Our results built upon minimal set of assumptions while cover a wide spectrum of adversarial perturbations including $\ell_p$ balls for any $p \ge 1$.

[1]  Aleksander Madry,et al.  Adversarially Robust Generalization Requires More Data , 2018, NeurIPS.

[2]  Daniel Cullina,et al.  Lower Bounds on Adversarial Robustness from Optimal Transport , 2019, NeurIPS.

[3]  Di He,et al.  Adversarially Robust Generalization Just Requires More Unlabeled Data , 2019, ArXiv.

[4]  Adel Javanmard,et al.  Precise Tradeoffs in Adversarial Training for Linear Regression , 2020, COLT.

[5]  P. Deb.  Finite Mixture Models , 2008 .

[6]  Po-Sen Huang,et al.  Are Labels Required for Improving Adversarial Robustness? , 2019, NeurIPS.

[7]  Stephen P. Boyd,et al.  Robust Fisher Discriminant Analysis , 2005, NIPS.

[8]  Martin J. Wainwright,et al.  High-Dimensional Statistics , 2019 .

[9]  K. Joossens.  Robust discriminant analysis , 2006 .

[10]  Shie Mannor,et al.  Robust Regression and Lasso , 2008, IEEE Transactions on Information Theory.

[11]  Larry A. Wasserman,et al.  Minimax Theory for High-dimensional Gaussian Mixtures with Sparse Mean Separation , 2013, NIPS.

[12]  Shie Mannor,et al.  Robustness and generalization , 2010, Machine Learning.

[13]  Yishay Mansour,et al.  Improved generalization bounds for robust learning , 2018, ALT.

[14]  Ilya P. Razenshteyn,et al.  Adversarial examples from computational constraints , 2018, ICML.

[15]  Aravindan Vijayaraghavan,et al.  On Robustness to Adversarial Examples and Polynomial Optimization , 2019, NeurIPS.

[16]  Po-Ling Loh,et al.  Adversarial Risk Bounds for Binary Classification via Function Transformation , 2018, ArXiv.

[17]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[18]  John Duchi,et al.  Understanding and Mitigating the Tradeoff Between Robustness and Accuracy , 2020, ICML.

[19]  Ananthram Swami,et al.  The Limitations of Deep Learning in Adversarial Settings , 2015, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[20]  Yin Tat Lee,et al.  Adversarial Examples from Cryptographic Pseudo-Random Generators , 2018, ArXiv.

[21]  Somesh Jha,et al.  Analyzing the Robustness of Nearest Neighbors to Adversarial Examples , 2017, ICML.

[22]  Ludwig Schmidt,et al.  Unlabeled Data Improves Adversarial Robustness , 2019, NeurIPS.

[23]  Linjun Zhang,et al.  High dimensional linear discriminant analysis: optimality, adaptive algorithm and missing data , 2018, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[24]  Michael I. Jordan,et al.  Theoretically Principled Trade-off between Robustness and Accuracy , 2019, ICML.

[25]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[26]  Prateek Mittal,et al.  PAC-learning in the presence of adversaries , 2018, NeurIPS.

[27]  Pradeep Ravikumar,et al.  Fast Classification Rates for High-dimensional Gaussian Generative Models , 2015, NIPS.

[28]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[29]  Shie Mannor,et al.  Robustness and Regularization of Support Vector Machines , 2008, J. Mach. Learn. Res..

[30]  Inderjit S. Dhillon,et al.  Towards Fast Computation of Certified Robustness for ReLU Networks , 2018, ICML.

[31]  Charles E. Heckler,et al.  Applied Multivariate Statistical Analysis , 2005, Technometrics.

[32]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[33]  Kannan Ramchandran,et al.  Rademacher Complexity for Adversarially Robust Generalization , 2018, ICML.

[34]  Pranjal Awasthi,et al.  Adversarial Learning Guarantees for Linear Hypotheses and Neural Networks , 2020, ICML.

[35]  Pradeep Ravikumar,et al.  Revisiting Adversarial Risk , 2018, AISTATS.

[36]  Nathan Srebro,et al.  VC Classes are Adversarially Robustly Learnable, but Only Improperly , 2019, COLT.

[37]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[38]  Aleksander Madry,et al.  Robustness May Be at Odds with Accuracy , 2018, ICLR.

[39]  Pradeep Ravikumar,et al.  Minimax Gaussian Classification & Clustering , 2017, AISTATS.