A Channel-level Pruning Strategy for Convolutional Layers in CNNs

A large number of parameters and computing operations of deep convolutional neural networks(CNNs) make it hinder to apply them to real-world scenarios. In this paper, we propose a channel-level pruning strategy for convolutional layers to reduce the number of parameters and computing operations in CNNs with no accuracy loss. First, we use “Squeeze-and-Excitation” block to extract the activation factors of each sample, which can be used to evaluate the importance of each channel. Second, we compute the overall weight of a specific channel by accumulating its activation factors generated by all training samples. Finally, we prune the redundant channels with low weight and thus yield compact network. On a colorectal pathology dataset, we reduce the number of channels and the convolution layer parameters by a factor of 5× and 21×, without any accuracy loss.

[1]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[2]  Yoshua Bengio,et al.  BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 , 2016, ArXiv.

[3]  Danilo Comminiello,et al.  Group sparse regularization for deep neural networks , 2016, Neurocomputing.

[4]  Hassan Foroosh,et al.  Sparse Convolutional Neural Networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Yurong Chen,et al.  Dynamic Network Surgery for Efficient DNNs , 2016, NIPS.

[6]  Yixin Chen,et al.  Compressing Neural Networks with the Hashing Trick , 2015, ICML.

[7]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[8]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[9]  Zhiqiang Shen,et al.  Learning Efficient Convolutional Networks through Network Slimming , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[10]  Hao Zhou,et al.  Less Is More: Towards Compact CNNs , 2016, ECCV.

[11]  Song Han,et al.  EIE: Efficient Inference Engine on Compressed Deep Neural Network , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[12]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[13]  Yiran Chen,et al.  Learning Structured Sparsity in Deep Neural Networks , 2016, NIPS.

[14]  Mark Sandler,et al.  The Power of Sparsity in Convolutional Neural Networks , 2017, ArXiv.

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).