Improved Branch and Bound for Neural Network Verification via Lagrangian Decomposition

We improve the scalability of Branch and Bound (BaB) algorithms for formally proving input-output properties of neural networks. First, we propose novel bounding algorithms based on Lagrangian Decomposition. Previous works have used off-the-shelf solvers to solve relaxations at each node of the BaB tree, or constructed weaker relaxations that can be solved efficiently, but lead to unnecessarily weak bounds. Our formulation restricts the optimization to a subspace of the dual domain that is guaranteed to contain the optimum, resulting in accelerated convergence. Furthermore, it allows for a massively parallel implementation, which is amenable to GPU acceleration via modern deep learning frameworks. Second, we present a novel activation-based branching strategy. By coupling an inexpensive heuristic with fast dual bounding, our branching scheme greatly reduces the size of the BaB tree compared to previous heuristic methods. Moreover, it performs competitively with a recent strategy based on learning algorithms, without its large offline training cost. Finally, we design a BaB framework, named Branch and Dual Network Bound (BaDNB), based on our novel bounding and branching algorithms. We show that BaDNB outperforms previous complete verification systems by a large margin, cutting average verification times by factors up to 50 on adversarial robustness properties.

[1]  Timon Gehr,et al.  An abstract domain for certifying neural networks , 2019, Proc. ACM Program. Lang..

[2]  Russ Tedrake,et al.  Evaluating Robustness of Neural Networks with Mixed Integer Programming , 2017, ICLR.

[3]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[4]  Claude Lemaréchal,et al.  Lagrangian Relaxation , 2000, Computational Combinatorial Optimization.

[5]  M. Pawan Kumar,et al.  Neural Network Branching for Neural Network Verification , 2019, ICLR.

[6]  Cho-Jui Hsieh,et al.  A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks , 2019, NeurIPS.

[7]  Francis R. Bach,et al.  Duality Between Subgradient and Conditional Gradient Methods , 2012, SIAM J. Optim..

[8]  Mislav Balunovic,et al.  Adversarial Training and Provable Defenses: Bridging the Gap , 2020, ICLR.

[9]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[10]  Taylor T. Johnson,et al.  Improved Geometric Path Enumeration for Verifying ReLU Neural Networks , 2020, CAV.

[11]  Pushmeet Kohli,et al.  A Dual Approach to Scalable Verification of Deep Networks , 2018, UAI.

[12]  Matthew Mirman,et al.  Differentiable Abstract Interpretation for Provably Robust Neural Networks , 2018, ICML.

[13]  Cho-Jui Hsieh,et al.  Efficient Neural Network Robustness Certification with General Activation Functions , 2018, NeurIPS.

[14]  Junfeng Yang,et al.  Efficient Formal Safety Analysis of Neural Networks , 2018, NeurIPS.

[15]  Timon Gehr,et al.  Boosting Robustness Certification of Neural Networks , 2018, ICLR.

[16]  Mark W. Schmidt,et al.  Block-Coordinate Frank-Wolfe Optimization for Structural SVMs , 2012, ICML.

[17]  Pushmeet Kohli,et al.  Branch and Bound for Piecewise Linear Neural Network Verification , 2020, J. Mach. Learn. Res..

[18]  P. Henriksen,et al.  Efficient Neural Network Verification via Adaptive Refinement and Adversarial Search , 2020, ECAI.

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Sheldon H. Jacobson,et al.  Branch-and-bound algorithms: A survey of recent advances in searching, branching, and pruning , 2016, Discret. Optim..

[21]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[22]  Inderjit S. Dhillon,et al.  Towards Fast Computation of Certified Robustness for ReLU Networks , 2018, ICML.

[23]  Weiming Xiang,et al.  Star-Based Reachability Analysis of Deep Neural Networks , 2019, FM.

[24]  Pushmeet Kohli,et al.  Adversarial Risk and the Dangers of Evaluating Against Weak Attacks , 2018, ICML.

[25]  Hanif D. Sherali,et al.  A Hierarchy of Relaxations and Convex Hull Characterizations for Mixed-integer Zero-one Programming Problems , 1994, Discret. Appl. Math..

[26]  Junfeng Yang,et al.  Formal Security Analysis of Neural Networks using Symbolic Intervals , 2018, USENIX Security Symposium.

[27]  Saverio Salzo,et al.  Inexact and accelerated proximal point algorithms , 2011 .

[28]  Matthew Mirman,et al.  Fast and Effective Robustness Certification , 2018, NeurIPS.

[29]  Monique Guignard-Spielberg,et al.  Lagrangean decomposition: A model yielding stronger lagrangean bounds , 1987, Math. Program..

[30]  Zaïd Harchaoui,et al.  Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice , 2017, J. Mach. Learn. Res..

[31]  Dusan M. Stipanovic,et al.  Fast Neural Network Verification via Shadow Prices , 2019, ArXiv.

[32]  David A. Wagner,et al.  Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples , 2018, ICML.

[33]  Pushmeet Kohli,et al.  Lagrangian Decomposition for Neural Network Verification , 2020, UAI.

[34]  Rüdiger Ehlers,et al.  Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks , 2017, ATVA.

[35]  Aditi Raghunathan,et al.  Semidefinite relaxations for certifying robustness to adversarial examples , 2018, NeurIPS.

[36]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[37]  Christian Tjandraatmadja,et al.  Strong mixed-integer programming formulations for trained neural networks , 2018, Mathematical Programming.

[38]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[39]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[40]  Timothy A. Mann,et al.  On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models , 2018, ArXiv.

[41]  Pushmeet Kohli,et al.  A Unified View of Piecewise Linear Neural Network Verification , 2017, NeurIPS.

[42]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[43]  Martin Vechev,et al.  Beyond the Single Neuron Convex Barrier for Neural Network Certification , 2019, NeurIPS.

[44]  J. Zico Kolter,et al.  Provable defenses against adversarial examples via the convex outer adversarial polytope , 2017, ICML.

[45]  Tobias Achterberg,et al.  Mixed Integer Programming: Analyzing 12 Years of Progress , 2013 .

[46]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.