Adversarial Training with Generated Data in High-Dimensional Regression: An Asymptotic Study

In recent years, studies such as \cite{carmon2019unlabeled,gowal2021improving,xing2022artificial} have demonstrated that incorporating additional real or generated data with pseudo-labels can enhance adversarial training through a two-stage training approach. In this paper, we perform a theoretical analysis of the asymptotic behavior of this method in high-dimensional linear regression. While a double-descent phenomenon can be observed in ridgeless training, with an appropriate $\mathcal{L}_2$ regularization, the two-stage adversarial training achieves a better performance. Finally, we derive a shortcut cross-validation formula specifically tailored for the two-stage training method.

[1]  Teng Zhang,et al.  Understanding Overfitting in Adversarial Training via Kernel Regression , 2023, ArXiv.

[2]  Min Lin,et al.  Better Diffusion Models Further Improve Adversarial Training , 2023, ICML.

[3]  Ruoyu Sun,et al.  Adversarial Rademacher Complexity of Deep Neural Networks , 2022, ArXiv.

[4]  Ruoyu Sun,et al.  Stability Analysis and Generalization Bounds of Adversarial Training , 2022, NeurIPS.

[5]  James Bailey,et al.  On the Convergence and Robustness of Adversarial Training , 2021, ICML.

[6]  Sven Gowal,et al.  Improving Robustness using Generated Data , 2021, NeurIPS.

[7]  Guang Cheng,et al.  Adversarially Robust Estimate and Risk Analysis in Linear Regression , 2020, AISTATS.

[8]  Jeffrey Pennington,et al.  Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition , 2020, NeurIPS.

[9]  J. Rocks,et al.  Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models , 2020, Physical review research.

[10]  Nicolas Flammarion,et al.  RobustBench: a standardized adversarial robustness benchmark , 2020, NeurIPS Datasets and Benchmarks.

[11]  Zhenyu Liao,et al.  Kernel regression in high dimension: Refined analysis beyond double descent , 2020, AISTATS.

[12]  Pradeep Ravikumar,et al.  Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification , 2020, ArXiv.

[13]  James Y. Zou,et al.  Improving Adversarial Robustness via Unlabeled Out-of-Domain Data , 2020, AISTATS.

[14]  Johannes Lederer,et al.  Statistical Guarantees for Regularized Neural Networks , 2020, Neural Networks.

[15]  Taiji Suzuki,et al.  Generalization of Two-layer Neural Networks: An Asymptotic Viewpoint , 2020, ICLR.

[16]  James Bailey,et al.  Improving Adversarial Robustness Requires Revisiting Misclassified Examples , 2020, ICLR.

[17]  Yisen Wang,et al.  Adversarial Weight Perturbation Helps Robust Generalization , 2020, NeurIPS.

[18]  G. Biroli,et al.  Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime , 2020, ICML.

[19]  Mohan S. Kankanhalli,et al.  Attacks Which Do Not Kill Training Make Adversarial Learning Stronger , 2020, ICML.

[20]  Amin Karbasi,et al.  The Curious Case of Adversarially Robust Models: More Data Can Help, Double Descend, or Hurt Generalization , 2020, UAI.

[21]  Adel Javanmard,et al.  Precise Tradeoffs in Adversarial Training for Linear Regression , 2020, COLT.

[22]  Amin Karbasi,et al.  More Data Can Expand the Generalization Gap Between Adversarially Robust and Standard Models , 2020, ICML.

[23]  Zhao Song,et al.  Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality , 2020, NeurIPS.

[24]  Ludwig Schmidt,et al.  Unlabeled Data Improves Adversarial Robustness , 2019, NeurIPS.

[25]  Aditi Raghunathan,et al.  Adversarial Training Can Hurt Generalization , 2019, ArXiv.

[26]  Amir Najafi,et al.  Robustness to Adversarial Perturbations in Learning from Incomplete Data , 2019, NeurIPS.

[27]  Andrea Montanari,et al.  Surprises in High-Dimensional Ridgeless Least Squares Interpolation , 2019, Annals of statistics.

[28]  Mikhail Belkin,et al.  Two models of double descent for weak features , 2019, SIAM J. Math. Data Sci..

[29]  Kimin Lee,et al.  Using Pre-Training Can Improve Model Robustness and Uncertainty , 2019, ICML.

[30]  Kannan Ramchandran,et al.  Rademacher Complexity for Adversarially Robust Generalization , 2018, ICML.

[31]  Dawn Xiaodong Song,et al.  Curriculum Adversarial Training , 2018, IJCAI.

[32]  John C. Duchi,et al.  Certifying Some Distributional Robustness with Principled Adversarial Training , 2017, ICLR.

[33]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[34]  Browne,et al.  Cross-Validation Methods. , 2000, Journal of mathematical psychology.

[35]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[36]  R. Dennis Cook,et al.  Cross-Validation of Regression Models , 1984 .

[37]  Qifan Song,et al.  Why Do Artificially Generated Data Help Adversarial Robustness , 2022, NeurIPS.

[38]  Zhanxing Zhu,et al.  Implicit Bias of Adversarial Training for Deep Neural Networks , 2022, ICLR.

[39]  Cho-Jui Hsieh,et al.  Convergence of Adversarial Training in Overparametrized Networks , 2019, ArXiv.