DeepCov: Coverage Guided Deep Learning Framework Fuzzing

The large-scale use of deep learning (DL) technology makes it important to ensure software security in DL systems. All DL systems depend on the DL framework to achieve development tasks. If any vulnerability exists in the DL framework, DL systems will contain potential security risks. In addition, existing DL framework testing methods typically use inconsistencies as the feedback and cannot effectively cover the entire layer, leading to inefficiencies in finding vulnerabilities.In this work, we propose DeepCov, a layer edge coverage based DL framework fuzzing system. DeepCov uses the mutation-based method to generate models that fully explore layer APIs of the DL framework. DeepCov detects three types of bugs, including inconsistency, NaN and crash bugs. Compared to the previous inconsistency guided tool LEMON, DeepCov considers the layer edge coverage metric and uses the multi-dimensional evaluation based seed selection and strategic layer addition operator. By analysing and utilizing the information from models, DeepCov generates mutated models with more API calls of layer types which eventually explore larger model layer space in the DL framework testing. To show the effectiveness of our approach, we evaluated DeepCov on five DL frameworks, i.e., TensorFlow, Pytorch, CNTK, Theano and MXNet. The results show that DeepCov achieves 5 times improvement in the number of tested layer types compared with LEMON. Moreover, 6 crash bugs, 11 unique inconsistency bugs, and a NaN bug were found.

[1]  Yu Liu,et al.  Enhancing the interoperability between deep learning frameworks by model conversion , 2020, ESEC/SIGSOFT FSE.

[2]  Ming Yan,et al.  Deep learning library testing via effective model generation , 2020, ESEC/SIGSOFT FSE.

[3]  Chao Shen,et al.  Audee: Automated Testing for Deep Learning Frameworks , 2020, 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[4]  Touseef Iqbal,et al.  The survey: Text generation models in deep learning , 2020, J. King Saud Univ. Comput. Inf. Sci..

[5]  Tiberiu T. Cocias,et al.  A survey of deep learning techniques for autonomous driving , 2019, J. Field Robotics.

[6]  Lin Tan,et al.  CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[7]  Chao Zhang,et al.  Fuzzing: a survey , 2018, Cybersecur..

[8]  Lei Ma,et al.  DeepMutation: Mutation Testing of Deep Learning Systems , 2018, 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE).

[9]  Björn Schuller,et al.  Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments , 2017 .

[10]  Tony Beltramelli,et al.  pix2code: Generating Code from a Graphical User Interface Screenshot , 2017, EICS.

[11]  Tao Xie,et al.  Multiple-Implementation Testing of Supervised Learning Software , 2016, AAAI Workshops.

[12]  Karthik Duraisamy,et al.  A paradigm for data-driven predictive modeling using field inversion and machine learning , 2016, J. Comput. Phys..

[13]  John Tran,et al.  cuDNN: Efficient Primitives for Deep Learning , 2014, ArXiv.

[14]  Adam Lipowski,et al.  Roulette-wheel selection via stochastic acceptance , 2011, ArXiv.