MuNN: Mutation Analysis of Neural Networks

Deep neural networks have made amazing progress in many areas over the past few years. After training deep neural networks, a common way is to build a set of test samples to evaluate the networks. However, test adequacy of deep neural networks is overlooked in both research and practice. This paper proposes MuNN, a mutation analysis method for (deep) neural networks, inspired by the success of mutation analysis in conventional software testing. Five mutation operators are designed on the characteristics of neural networks. The well known dataset MNIST has been used in our experiment. We study two research questions: (1) How does mutation affect neural networks; (2) How does neural depth affect mutation analysis. The experimental results show that mutation analysis of neural networks has strong domain characteristics. This indicates that domain mutation operators are needed to enhance mutation analysis. The experimental results also show that the mutation effects are gradually weakened with the deepening of neurons. We need to design new mutation mechanism for deep neural networks. We believe that mutation analysis can not only be used to measure test adequacy, but also be used to understand neural network operation.

[1]  Francisco Herrera,et al.  An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics , 2013, Inf. Sci..

[2]  Junfeng Yang,et al.  DeepXplore: Automated Whitebox Testing of Deep Learning Systems , 2019, GETMBL.

[3]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[4]  Ronald M. Summers,et al.  Deep Learning in Medical Imaging: Overview and Future Promise of an Exciting New Technique , 2016 .

[5]  Geoffrey Zweig,et al.  Achieving Human Parity in Conversational Speech Recognition , 2016, ArXiv.

[6]  Baowen Xu,et al.  A Novel Method of Mutation Clustering Based on Domain Analysis , 2009, SEKE.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Timothy A. Budd,et al.  Program Testing by Specification Mutation , 1985, Comput. Lang..

[9]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[10]  Honglak Lee,et al.  Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units , 2016, ICML.

[11]  Axel Hollmann,et al.  Positive and Negative Testing with Mutation-Driven Model Checking , 2008, GI Jahrestagung.

[12]  Baowen Xu,et al.  A New Mutation Analysis Method for Testing Java Exception Handling , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[13]  Zhenlong Yuan,et al.  Droid-Sec: deep learning in android malware detection , 2015, SIGCOMM 2015.

[14]  Yurong Chen,et al.  Dynamic Network Surgery for Efficient DNNs , 2016, NIPS.

[15]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[16]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[17]  Ajmal Mian,et al.  Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey , 2018, IEEE Access.

[18]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[19]  Richard G. Hamlet,et al.  Testing Programs with the Aid of a Compiler , 1977, IEEE Transactions on Software Engineering.

[20]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[21]  Yuming Zhou,et al.  Mutation Selection: Some Could be Better than All , 2011, EAST.