Algorithms for Verifying Deep Neural Networks

Deep neural networks are widely used for nonlinear function approximation with applications ranging from computer vision to control. Although these networks involve the composition of simple arithmetic operations, it can be very challenging to verify whether a particular network satisfies certain input-output properties. This article surveys methods that have emerged recently for soundly verifying such properties. These methods borrow insights from reachability analysis, optimization, and search. We discuss fundamental differences and connections between existing algorithms. In addition, we provide pedagogical implementations of existing methods and compare them on a set of benchmark problems.

[1]  Christian Tjandraatmadja,et al.  Strong mixed-integer programming formulations for trained neural networks , 2018, Mathematical Programming.

[2]  Justin Emile Gottschlich,et al.  An Abstraction-Based Framework for Neural Network Verification , 2019, CAV.

[3]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2017, SOSP.

[4]  Weiming Xiang,et al.  Verification for Machine Learning, Autonomy, and Neural Networks Survey , 2018, ArXiv.

[5]  Yasser Shoukry,et al.  Effective Formal Verification of Neural Networks using the Geometry of Linear Regions , 2020, ArXiv.

[6]  Weiming Xiang,et al.  Reachable Set Estimation and Safety Verification for Piecewise Linear Systems with Neural Network Controllers , 2018, 2018 Annual American Control Conference (ACC).

[7]  Weiming Xiang,et al.  Output Reachable Set Estimation and Verification for Multilayer Neural Networks , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[8]  Sergiy Bogomolov,et al.  JuliaReach: a toolbox for set-based reachability , 2019, HSCC.

[9]  Iain Dunning,et al.  JuMP: A Modeling Language for Mathematical Optimization , 2015, SIAM Rev..

[10]  Dusan M. Stipanovic,et al.  Fast Neural Network Verification via Shadow Prices , 2019, ArXiv.

[11]  M. Pawan Kumar,et al.  Neural Network Branching for Neural Network Verification , 2019, ICLR.

[12]  Alessio Lomuscio,et al.  Verification of RNN-Based Neural Agent-Environment Systems , 2019, AAAI.

[13]  Mykel J. Kochenderfer,et al.  Deep Neural Network Compression for Aircraft Collision Avoidance Systems , 2018, Journal of Guidance, Control, and Dynamics.

[14]  Matthew Mirman,et al.  Fast and Effective Robustness Certification , 2018, NeurIPS.

[15]  Cho-Jui Hsieh,et al.  RecurJac: An Efficient Recursive Algorithm for Bounding Jacobian Matrix of Neural Networks and Its Applications , 2018, AAAI.

[16]  Taylor T. Johnson,et al.  Improved Geometric Path Enumeration for Verifying ReLU Neural Networks , 2020, CAV.

[17]  Leonid Ryzhyk,et al.  Verifying Properties of Binarized Deep Neural Networks , 2017, AAAI.

[18]  Pushmeet Kohli,et al.  A Dual Approach to Scalable Verification of Deep Networks , 2018, UAI.

[19]  Daniel Kroening,et al.  Testing Deep Neural Networks , 2018, ArXiv.

[20]  Weiming Xiang,et al.  Parallelizable Reachability Analysis Algorithms for Feed-Forward Neural Networks , 2019, 2019 IEEE/ACM 7th International Conference on Formal Methods in Software Engineering (FormaliSE).

[21]  Weiming Xiang,et al.  Reachability Analysis for Feed-Forward Neural Networks using Face Lattices , 2020, ArXiv.

[22]  Liqian Chen,et al.  Analyzing Deep Neural Networks with Symbolic Propagation: Towards Higher Precision and Faster Verification , 2019, SAS.

[23]  Antonio Criminisi,et al.  Measuring Neural Net Robustness with Constraints , 2016, NIPS.

[24]  Yunong Zhang,et al.  Dual Neural Network , 2013 .

[25]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[26]  Chung-Hao Huang,et al.  Verification of Binarized Neural Networks via Inter-neuron Factoring - (Short Paper) , 2017, VSTTE.

[27]  Mihai Surdeanu,et al.  The Stanford CoreNLP Natural Language Processing Toolkit , 2014, ACL.

[28]  Cesare Tinelli,et al.  Satisfiability Modulo Theories , 2021, Handbook of Satisfiability.

[29]  P. Henriksen,et al.  Efficient Neural Network Verification via Adaptive Refinement and Adversarial Search , 2020, ECAI.

[30]  Pushmeet Kohli,et al.  Lagrangian Decomposition for Neural Network Verification , 2020, UAI.

[31]  Rüdiger Ehlers,et al.  Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks , 2017, ATVA.

[32]  Swarat Chaudhuri,et al.  AI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[33]  Matthew Mirman,et al.  Differentiable Abstract Interpretation for Provably Robust Neural Networks , 2018, ICML.

[34]  Weiming Xiang,et al.  Reachable Set Computation and Safety Verification for Neural Networks with ReLU Activations , 2017, ArXiv.

[35]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[36]  J Hayhurst Kelly,et al.  A Practical Tutorial on Modified Condition/Decision Coverage , 2001 .

[37]  Alessio Lomuscio,et al.  Reachability Analysis for Neural Agent-Environment Systems , 2018, KR.

[38]  Weiming Xiang,et al.  Specification-Guided Safety Verification for Feedforward Neural Networks , 2018, ArXiv.

[39]  Ashish Tiwari,et al.  Output Range Analysis for Deep Neural Networks , 2017, ArXiv.

[40]  Manfred Morari,et al.  Safety Verification and Robustness Analysis of Neural Networks via Quadratic Constraints and Semidefinite Programming , 2019, ArXiv.

[41]  Julian D. Olden,et al.  Illuminating the “black box”: a randomization approach for understanding variable contributions in artificial neural networks , 2002 .

[42]  Junfeng Yang,et al.  Efficient Formal Safety Analysis of Neural Networks , 2018, NeurIPS.

[43]  Aditi Raghunathan,et al.  Certified Defenses against Adversarial Examples , 2018, ICLR.

[44]  Jinfeng Yi,et al.  Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach , 2018, ICLR.

[45]  Cho-Jui Hsieh,et al.  Efficient Neural Network Robustness Certification with General Activation Functions , 2018, NeurIPS.

[46]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Alan Edelman,et al.  Julia: A Fresh Approach to Numerical Computing , 2014, SIAM Rev..

[48]  Pushmeet Kohli,et al.  Branch and Bound for Piecewise Linear Neural Network Verification , 2020, J. Mach. Learn. Res..

[49]  Cho-Jui Hsieh,et al.  A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks , 2019, NeurIPS.

[50]  Inderjit S. Dhillon,et al.  Towards Fast Computation of Certified Robustness for ReLU Networks , 2018, ICML.

[51]  Weiming Xiang,et al.  Star-Based Reachability Analysis of Deep Neural Networks , 2019, FM.

[52]  Yunong Zhang,et al.  Repetitive Motion Planning and Control of Redundant Robot Manipulators , 2013 .

[53]  Zahra Rahimi Afzal,et al.  Abstraction based Output Range Analysis for Neural Networks , 2020, NeurIPS.

[54]  Mykel J. Kochenderfer,et al.  The Marabou Framework for Verification and Analysis of Deep Neural Networks , 2019, CAV.

[55]  Junfeng Yang,et al.  Formal Security Analysis of Neural Networks using Symbolic Intervals , 2018, USENIX Security Symposium.

[56]  Pushmeet Kohli,et al.  A Unified View of Piecewise Linear Neural Network Verification , 2017, NeurIPS.

[57]  Weiming Xiang,et al.  NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems , 2020, CAV.

[58]  Martin Vechev,et al.  Beyond the Single Neuron Convex Barrier for Neural Network Certification , 2019, NeurIPS.

[59]  Yann Le Cun,et al.  A Theoretical Framework for Back-Propagation , 1988 .

[60]  J. Zico Kolter,et al.  Provable defenses against adversarial examples via the convex outer adversarial polytope , 2017, ICML.

[61]  Timon Gehr,et al.  Boosting Robustness Certification of Neural Networks , 2018, ICLR.

[62]  Russ Tedrake,et al.  Evaluating Robustness of Neural Networks with Mixed Integer Programming , 2017, ICLR.

[63]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[64]  Ashish Tiwari,et al.  Learning and Verification of Feedback Control Systems using Feedforward Neural Networks , 2018, ADHS.

[65]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[66]  Alessio Lomuscio,et al.  An approach to reachability analysis for feed-forward ReLU neural networks , 2017, ArXiv.

[67]  E. Yaz.  Linear Matrix Inequalities In System And Control Theory , 1998, Proceedings of the IEEE.

[68]  Ananthram Swami,et al.  The Limitations of Deep Learning in Adversarial Settings , 2015, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[69]  Somayeh Sojoudi,et al.  Tightened Convex Relaxations for Neural Network Robustness Certification , 2020, 2020 59th IEEE Conference on Decision and Control (CDC).

[70]  Chih-Hong Cheng,et al.  Verification of Binarized Neural Networks , 2017, ArXiv.

[71]  Luca Pulina,et al.  Automated Verification of Neural Networks: Advances, Challenges and Perspectives , 2018, ArXiv.

[72]  Timon Gehr,et al.  An abstract domain for certifying neural networks , 2019, Proc. ACM Program. Lang..

[73]  Weiming Xiang,et al.  Verification of Deep Convolutional Neural Networks Using ImageStars , 2020, CAV.