TAPS: Connecting Certified and Adversarial Training

Training certifiably robust neural networks remains a notoriously hard problem. On one side, adversarial training optimizes under-approximations of the worst-case loss, which leads to insufficient regularization for certification, while on the other, sound certified training methods optimize loose over-approximations, leading to over-regularization and poor (standard) accuracy. In this work we propose TAPS, an (unsound) certified training method that combines IBP and PGD training to yield precise, although not necessarily sound, worst-case loss approximations, reducing over-regularization and increasing certified and standard accuracies. Empirically, TAPS achieves a new state-of-the-art in many settings, e.g., reaching a certified accuracy of $22\%$ on TinyImageNet for $\ell_\infty$-perturbations with radius $\epsilon=1/255$.

[1]  Martin T. Vechev,et al.  Certified Training: Small Boxes are All You Need , 2022, ICLR.

[2]  Liwei Wang,et al.  Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective , 2022, NeurIPS.

[3]  J. Z. Kolter,et al.  General Cutting Planes for Bound-Propagation-Based Neural Network Verification , 2022, NeurIPS.

[4]  M. P. Kumar,et al.  IBP Regularization for Verified Adversarial Robustness via Branch-and-Bound , 2022, ArXiv.

[5]  Martin T. Vechev,et al.  Complete Verification via Multi-Neuron Relaxation Guided Branch-and-Bound , 2022, ICLR.

[6]  Di He,et al.  Boosting the Certified Robustness of L-infinity Distance Nets , 2021, ICLR.

[7]  Cho-Jui Hsieh,et al.  Fast Certified Robust Training with Short Warmup , 2021, NeurIPS.

[8]  Cho-Jui Hsieh,et al.  Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Neural Network Robustness Verification , 2021, NeurIPS.

[9]  Mark Niklas Müller,et al.  PRIMA: general and precise neural network certification via scalable convex hull approximations , 2021, Proc. ACM Program. Lang..

[10]  Di He,et al.  Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons , 2021, ICML.

[11]  Ian Goodfellow,et al.  Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming , 2020, NeurIPS.

[12]  Mislav Balunovic,et al.  Adversarial Training and Provable Defenses: Bridging the Gap , 2020, ICLR.

[13]  Matthias Hein,et al.  Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks , 2020, ICML.

[14]  Cho-Jui Hsieh,et al.  Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond , 2020, NeurIPS.

[15]  Florian Tramèr,et al.  On Adaptive Attacks to Adversarial Example Defenses , 2020, NeurIPS.

[16]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[17]  Pushmeet Kohli,et al.  Branch and Bound for Piecewise Linear Neural Network Verification , 2019, J. Mach. Learn. Res..

[18]  Cho-Jui Hsieh,et al.  Towards Stable and Efficient Training of Verifiably Robust Neural Networks , 2019, ICLR.

[19]  Mislav Balunovic,et al.  DL2: Training and Querying Neural Networks with Logic , 2019, ICML.

[20]  J. Zico Kolter,et al.  Certified Adversarial Robustness via Randomized Smoothing , 2019, ICML.

[21]  Timon Gehr,et al.  An abstract domain for certifying neural networks , 2019, Proc. ACM Program. Lang..

[22]  Cho-Jui Hsieh,et al.  Efficient Neural Network Robustness Certification with General Activation Functions , 2018, NeurIPS.

[23]  Timothy A. Mann,et al.  On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models , 2018, ArXiv.

[24]  Cem Anil,et al.  Sorting out Lipschitz function approximation , 2018, ICML.

[25]  L. Carin,et al.  Certified Adversarial Robustness with Additive Noise , 2018, NeurIPS.

[26]  Matthew Mirman,et al.  Differentiable Abstract Interpretation for Provably Robust Neural Networks , 2018, ICML.

[27]  J. Zico Kolter,et al.  Scaling provable adversarial defenses , 2018, NeurIPS.

[28]  Suman Jana,et al.  Certified Robustness to Adversarial Examples with Differential Privacy , 2018, 2019 IEEE Symposium on Security and Privacy (SP).

[29]  Aditi Raghunathan,et al.  Certified Defenses against Adversarial Examples , 2018, ICLR.

[30]  Russ Tedrake,et al.  Evaluating Robustness of Neural Networks with Mixed Integer Programming , 2017, ICLR.

[31]  J. Zico Kolter,et al.  Provable defenses against adversarial examples via the convex outer adversarial polytope , 2017, ICML.

[32]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[33]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[34]  Fabio Roli,et al.  Evasion Attacks against Machine Learning at Test Time , 2013, ECML/PKDD.

[35]  Harkirat Singh Behl,et al.  Scaling the Convex Barrier with Active Sets , 2021, ICLR.

[36]  Bai Li.  Certified Adversarial Robustness with Additive Noise , 2019 .

[37]  Matthew Mirman,et al.  Fast and Effective Robustness Certification , 2018, NeurIPS.

[38]  Ya Le,et al.  Tiny ImageNet Visual Recognition Challenge , 2015 .

[39]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .