The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results

This report summarizes the second International Verification of Neural Networks Competition (VNN-COMP 2021), held as a part of the 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems that was collocated with the 33rd International Conference on Computer-Aided Verification (CAV). Twelve teams participated in this competition. The goal of the competition is to provide an objective comparison of the state-of-the-art methods in neural network verification, in terms of scalability and speed. Along this line, we used standard formats (ONNX for neural networks and VNNLIB for specifications), standard hardware (all tools are run by the organizers on AWS), and tool parameters provided by the tool authors. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this competition.

[1]  Taylor T. Johnson,et al.  First three years of the international verification of neural networks competition (VNN-COMP) , 2023, International Journal on Software Tools for Technology Transfer.

[2]  Alessio Lomuscio,et al.  DEEPSPLIT: An Efficient Splitting Method for Neural Network Verification via Indirect Effect Analysis , 2021, IJCAI.

[3]  Alessio Lomuscio,et al.  Towards Scalable Complete Verification of Relu Neural Networks via Dependency-based Branching , 2021, IJCAI.

[4]  Florian Jaeckle,et al.  Neural Network Branch-and-Bound for Neural Network Verification , 2021, ArXiv.

[5]  Florian Jaeckle,et al.  Generating Adversarial Examples with Graph Neural Networks , 2021, UAI.

[6]  Matthew B. Dwyer,et al.  DNNV: A Framework for Deep Neural Network Verification , 2021, CAV.

[7]  Matthew B. Dwyer,et al.  Reducing DNN Properties to Enable Falsification with Adversarial Attacks , 2021, 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE).

[8]  Pushmeet Kohli,et al.  Improved Branch and Bound for Neural Network Verification via Lagrangian Decomposition , 2021, ArXiv.

[9]  Philip H. S. Torr,et al.  Scaling the Convex Barrier with Sparse Dual Algorithms , 2021, ArXiv.

[10]  Yihan Wang,et al.  Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers , 2020, ICLR.

[11]  Mac Schwager,et al.  Reachable Polyhedral Marching (RPM): A Safety Verification Algorithm for Robotic Systems with Deep Neural Network Components , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[12]  Clark W. Barrett,et al.  An SMT-Based Approach for Verifying Binarized Neural Networks , 2020, TACAS.

[13]  Martin T. Vechev,et al.  Scaling Polyhedral Neural Network Verification on GPUs , 2020, MLSys.

[14]  T. Noll,et al.  Debona: Decoupled Boundary Network Analysis for Tighter Bounds and Faster Adversarial Robustness Proofs , 2020, ArXiv.

[15]  Taylor T. Johnson,et al.  Improved Geometric Path Enumeration for Verifying ReLU Neural Networks , 2020, CAV.

[16]  Mislav Balunovic,et al.  Adversarial Training and Provable Defenses: Bridging the Gap , 2020, ICLR.

[17]  Corina S. Pasareanu,et al.  Parallelization Techniques for Verifying Neural Networks , 2020, 2020 Formal Methods in Computer Aided Design (FMCAD).

[18]  Weiming Xiang,et al.  NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems , 2020, CAV.

[19]  Taylor T. Johnson,et al.  Verification of Deep Convolutional Neural Networks Using ImageStars , 2020, CAV.

[20]  Cho-Jui Hsieh,et al.  Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond , 2020, NeurIPS.

[21]  Pushmeet Kohli,et al.  Lagrangian Decomposition for Neural Network Verification , 2020, UAI.

[22]  M. Pawan Kumar,et al.  Neural Network Branching for Neural Network Verification , 2019, ICLR.

[23]  Weiming Xiang,et al.  Star-Based Reachability Analysis of Deep Neural Networks , 2019, FM.

[24]  Pushmeet Kohli,et al.  Branch and Bound for Piecewise Linear Neural Network Verification , 2019, J. Mach. Learn. Res..

[25]  Mykel J. Kochenderfer,et al.  The Marabou Framework for Verification and Analysis of Deep Neural Networks , 2019, CAV.

[26]  Weiming Xiang,et al.  Parallelizable Reachability Analysis Algorithms for Feed-Forward Neural Networks , 2019, 2019 IEEE/ACM 7th International Conference on Formal Methods in Software Engineering (FormaliSE).

[27]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[28]  Mykel J. Kochenderfer,et al.  Algorithms for Verifying Deep Neural Networks , 2019, Found. Trends Optim..

[29]  Cho-Jui Hsieh,et al.  A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks , 2019, NeurIPS.

[30]  Timon Gehr,et al.  An abstract domain for certifying neural networks , 2019, Proc. ACM Program. Lang..

[31]  Christian Tjandraatmadja,et al.  Strong mixed-integer programming formulations for trained neural networks , 2018, Mathematical Programming.

[32]  Cho-Jui Hsieh,et al.  Efficient Neural Network Robustness Certification with General Activation Functions , 2018, NeurIPS.

[33]  Timon Gehr,et al.  Boosting Robustness Certification of Neural Networks , 2018, ICLR.

[34]  Pushmeet Kohli,et al.  A Dual Approach to Scalable Verification of Deep Networks , 2018, UAI.

[35]  Ed H. Chi,et al.  The Case for Learned Index Structures , 2017, SIGMOD Conference.

[36]  Russ Tedrake,et al.  Evaluating Robustness of Neural Networks with Mixed Integer Programming , 2017, ICLR.

[37]  J. Zico Kolter,et al.  Provable defenses against adversarial examples via the convex outer adversarial polytope , 2017, ICML.

[38]  Pushmeet Kohli,et al.  A Unified View of Piecewise Linear Neural Network Verification , 2017, NeurIPS.

[39]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[40]  Jun Zhu,et al.  Boosting Adversarial Attacks with Momentum , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Weiming Xiang,et al.  Output Reachable Set Estimation and Verification for Multilayer Neural Networks , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[42]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[43]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[44]  Markus Püschel,et al.  Fast polyhedra abstract domain , 2017, POPL.

[45]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Cho-Jui Hsieh,et al.  Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification , 2021, ArXiv.

[47]  Martin T. Vechev,et al.  PRIMA: Precise and General Neural Network Certification via Multi-Neuron Convex Relaxations , 2021 .

[48]  Stanley Bak,et al.  nnenum: Verification of ReLU Neural Networks with Optimized Abstraction Refinement , 2021, NFM.

[49]  Harkirat Singh Behl,et al.  Scaling the Convex Barrier with Active Sets , 2021, ICLR.

[50]  P. Henriksen,et al.  Efficient Neural Network Verification via Adaptive Refinement and Adversarial Search , 2020, ECAI.

[51]  Martin Vechev,et al.  Beyond the Single Neuron Convex Barrier for Neural Network Certification , 2019, NeurIPS.

[52]  Matthew Mirman,et al.  Fast and Effective Robustness Certification , 2018, NeurIPS.