Exploiting Sparsity for Neural Network Verification

The problem of verifying the properties of a neural network has never been more important. This task is often done by bounding the activation functions in the network. Some approaches are more conservative than others and in general there is a trade-off between complexity and conservativeness. There has been significant progress to improve the efficiency and the accuracy of these methods. We investigate the sparsity that arises in a recently proposed semi-definite programming framework to verify a fully connected feed-forward neural network. We show that due to the intrinsic cascading structure of the neural network, the constraint matrices in the semi-definite program form a block-arrow pattern and satisfy conditions for chordal sparsity. We reformulate and implement the optimisation problem, showing a significant speed-up in computation, without sacrificing solution accuracy.

[1]  Jose Javier Gonzalez Ortiz,et al.  What is the State of Neural Network Pruning? , 2020, MLSys.

[2]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Aditi Raghunathan,et al.  Semidefinite relaxations for certifying robustness to adversarial examples , 2018, NeurIPS.

[4]  Antonis Papachristodoulou,et al.  Chordal sparsity, decomposing SDPs and the Lyapunov equation , 2014, 2014 American Control Conference.

[5]  Ian Goodfellow,et al.  Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming , 2020, NeurIPS.

[6]  Cho-Jui Hsieh,et al.  A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks , 2019, NeurIPS.

[7]  Yang Zheng,et al.  Chordal sparsity in control and optimization of large-scale systems , 2019 .

[8]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[9]  Taylor T. Johnson,et al.  ARCH-COMP19 Category Report: Artificial Intelligence and Neural Network Control Systems (AINNCS) for Continuous and Hybrid Systems Plants , 2019, ARCH@CPSIoTWeek.

[10]  George J. Pappas,et al.  Reach-SDP: Reachability Analysis of Closed-Loop Systems with Neural Network Controllers via Semidefinite Programming , 2020, 2020 59th IEEE Conference on Decision and Control (CDC).

[11]  George J. Pappas,et al.  Learning Lyapunov Functions for Piecewise Affine Systems with Neural Network Controllers , 2020 .

[12]  Manfred Morari,et al.  Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming , 2019, ArXiv.

[13]  Martin Vechev,et al.  Beyond the Single Neuron Convex Barrier for Neural Network Certification , 2019, NeurIPS.

[14]  Yang Zheng,et al.  Scalable Design of Structured Controllers Using Chordal Decomposition , 2018, IEEE Transactions on Automatic Control.

[15]  Masakazu Kojima,et al.  Exploiting sparsity in linear and nonlinear matrix inequalities via positive semidefinite matrix completion , 2011, Math. Program..

[16]  Yifan Sun,et al.  Decomposition in Conic Optimization with Partially Separable Structure , 2013, SIAM J. Optim..

[17]  Naonori Kakimura,et al.  A direct proof for the matrix decomposition of chordal-structured positive semidefinite matrices , 2010 .

[18]  J. Lofberg,et al.  YALMIP : a toolbox for modeling and optimization in MATLAB , 2004, 2004 IEEE International Conference on Robotics and Automation (IEEE Cat. No.04CH37508).

[19]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[20]  David A. Wagner,et al.  Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples , 2018, ICML.

[21]  Murat Arcak,et al.  Stability Analysis using Quadratic Constraints for Systems with Neural Network Controllers , 2020, ArXiv.

[22]  Eduardo D. Sontag,et al.  Neural Networks for Control , 1993 .

[23]  Benjamin Recht,et al.  A Tour of Reinforcement Learning: The View from Continuous Control , 2018, Annu. Rev. Control. Robotics Auton. Syst..

[24]  Juan Pablo Vielma,et al.  The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verification , 2020, NeurIPS.

[25]  Pushmeet Kohli,et al.  A Dual Approach to Scalable Verification of Deep Networks , 2018, UAI.

[26]  Cha Zhang,et al.  Ensemble Machine Learning: Methods and Applications , 2012 .

[27]  Paul Rolland,et al.  Lipschitz constant estimation of Neural Networks via sparse polynomial optimization , 2020, ICLR.

[28]  L. Rodman,et al.  Positive semidefinite matrices with a given sparsity pattern , 1988 .

[29]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .