Verification of Binarized Neural Networks

We study the problem of formal verification of Binarized Neural Networks (BNN), which have recently been proposed as a power-efficient alternative to more traditional learning networks. More precisely, given a trained BNN and a relation between possible inputs and outputs of this BNN, we develop verification procedures for establishing that the BNN indeed meets this specification for all possible inputs. For solving the verification problem of BNNs we build on well-known methods for hardware verification.The BNN verification problem is first encoded as a combinational miter. In a second step this miter is then transformed into a corresponding propositional satisfiability (SAT) problem. The main contributions of this paper are a number of essential optimizations for making this approach to BNN verification scalable. First, we provide a transformation on fully conntected BNNs for reducing the order of the number of bitwise operations in each layer of the BNN from quadratic to linear. Second, we are identifying redundant computations in a BNN based on {\em optimal factoring} techniques, and we provide transformations on BNNs for avoiding these multiple computations. We prove that the problem of optimal factoring is NP-hard, and we design efficient search procedures for generating approximate solutions of the optimal factoring problem. Third, we design a compositional verification procedure for analyzing each layer of a BNN separately, and for iteratively combining and refining local verification results. We experimentally demonstrate the scalability of our verification techniques to moderately-sized BNNs for embedded applications with thousands of neurons and inputs.

[1]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[2]  Armin Biere,et al.  PicoSAT Essentials , 2008, J. Satisf. Boolean Model. Comput..

[3]  Fernando A. Mujica,et al.  An Empirical Evaluation of Deep Learning on Highway Driving , 2015, ArXiv.

[4]  Alois Knoll,et al.  Deep neural networks for Markovian interactive scene prediction in highway scenarios , 2017, 2017 IEEE Intelligent Vehicles Symposium (IV).

[5]  Seyed-Mohsen Moosavi-Dezfooli,et al.  Universal Adversarial Perturbations , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Johann Glaser,et al.  Yosys-A Free Verilog Synthesis Suite , 2013 .

[7]  Jianxiong Xiao,et al.  DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[8]  Robert K. Brayton,et al.  ABC: An Academic Industrial-Strength Verification Tool , 2010, CAV.

[9]  René Peeters,et al.  The maximum edge biclique problem is NP-complete , 2003, Discret. Appl. Math..

[10]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[11]  Rüdiger Ehlers,et al.  Formal Verification of Piece-Wise Linear Feed-Forward Neural Networks , 2017, ATVA.

[12]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[13]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[14]  Johannes Stallkamp,et al.  The German Traffic Sign Recognition Benchmark: A multi-class classification competition , 2011, The 2011 International Joint Conference on Neural Networks.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[17]  Paris Smaragdis,et al.  Bitwise Neural Networks , 2016, ArXiv.

[18]  Alessio Lomuscio,et al.  An approach to reachability analysis for feed-forward ReLU neural networks , 2017, ArXiv.

[19]  Chih-Hong Cheng,et al.  Maximum Resilience of Artificial Neural Networks , 2017, ATVA.

[20]  Luca Pulina,et al.  An Abstraction-Refinement Approach to Verification of Artificial Neural Networks , 2010, CAV.

[21]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[22]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[23]  Wei Zhan,et al.  A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning , 2017, Volume 3: Modeling and Validation; Multi-Agent and Networked Systems; Path Planning and Motion Control; Tracking Control Systems; Unmanned Aerial Vehicles (UAVs) and Application; Unmanned Ground and Aerial Vehicles; Vibration in Mechanical Systems; Vibrat.

[24]  Mykel J. Kochenderfer,et al.  Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks , 2017, CAV.

[25]  S. Bhattacharyya,et al.  Certification considerations for adaptive systems , 2015, 2015 International Conference on Unmanned Aircraft Systems (ICUAS).