Loss Constrains Added Squeeze and Excitation Blocks for Pruning Deep Neural Networks

Deep neural networks are proved to be very effective to solve problems on image classification, object detection and segmentation. However, in cases where only limited hardware is acquired, it may be a problem to deploy big models with excellent performance as they are sometimes calculation consuming. To overcome the limits on power, memory and calculation, channel pruning is proposed to compress the model in channel wise and soon become a common approach to have big models compressed. Generally, pruning is a three-stage pipeline containing training, pruning and finetuning. In this work, we come up with a new pruning approach that needs no finetuning. The major idea is extracting channel saliences by squeeze and excitation block and pushing the salience to either 0 or 1 by a sin-based function. Then take the salience as criteria for pruning. As the criteria of our approach is activation rather than trainable parameter, finetuning is not necessary in our pruning strategy which make the pruning process more stable and time saving. Experiment on flowers demonstrates our new designed pruning method is effective on reducing the model scale while maintaining the overall accuracy.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[5]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Xiangyu Zhang,et al.  Channel Pruning for Accelerating Very Deep Neural Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[7]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  W. Marsden.  I and J , 2012 .

[9]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[10]  Zhiqiang Shen,et al.  Learning Efficient Convolutional Networks through Network Slimming , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[12]  Cheng-Zhong Xu,et al.  Dynamic Channel Pruning: Feature Boosting and Suppression , 2018, ICLR.

[13]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[14]  Timo Aila,et al.  Pruning Convolutional Neural Networks for Resource Efficient Transfer Learning , 2016, ArXiv.

[15]  Xiangyu Zhang,et al.  MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).