Providing clear pruning threshold: A novel CNN pruning method via L0 regularisation

Network pruning is a significant way to improve the practicability of convolution neural networks (CNNs) by removing the redundant structure of the network model. However, in most of the existing network pruning methods l 1 or l 2 regularisation is applied to parameter matrices and the manual selection of pruning threshold is difficult and labor-intensive. A novel CNNs network pruning method via l 0 regularisation is proposed, which adopts l 0 regularisation to expand the saliency gap between neurons. A half-quadratic splitting (HQS) based iterative algorithm is put forward to calculate the approximation solution of l 0 regularisation, which makes the joint optimisation problem of regularisation term and training loss function can be solved by various gradient-based algorithms. Meanwhile, a hyperparameters selection method is designed to make most of the hyperparameters in the algorithm can be determined by examining the pre-trained model. The results of experiments on MNIST, Fashion-MNIST and CIFAR100 show that the proposed method can provide a much clearer pruning threshold by widening the saliency gap, and achieve a similar or even better compression performance, compared with the state-of-the-art studies.

[1]  Xuelong Li,et al.  Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning , 2019, ArXiv.

[2]  Pavlo Molchanov,et al.  Importance Estimation for Neural Network Pruning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Jungong Han,et al.  Approximated Oracle Filter Pruning for Destructive CNN Width Optimization , 2019, ICML.

[4]  Ping Liu,et al.  Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Alessandro Rozza,et al.  Automated Pruning for Deep Neural Network Compression , 2017, 2018 24th International Conference on Pattern Recognition (ICPR).

[6]  Max Welling,et al.  Learning Sparse Neural Networks through L0 Regularization , 2017, ICLR.

[7]  Zhiqiang Shen,et al.  Learning Efficient Convolutional Networks through Network Slimming , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Jianxin Wu,et al.  ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[9]  Xin Dong,et al.  Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon , 2017, NIPS.

[10]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Timo Aila,et al.  Pruning Convolutional Neural Networks for Resource Efficient Inference , 2016, ICLR.

[12]  Hanan Samet,et al.  Pruning Filters for Efficient ConvNets , 2016, ICLR.

[13]  Justin Romberg,et al.  Net-Trim: A Layer-wise Convex Pruning of Deep Neural Networks , 2016, ArXiv.

[14]  Yiran Chen,et al.  Learning Structured Sparsity in Deep Neural Networks , 2016, NIPS.

[15]  Rui Peng,et al.  Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures , 2016, ArXiv.

[16]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Yike Guo,et al.  DropNeuron: Simplifying the Structure of Deep Neural Networks , 2016, ArXiv.

[18]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[19]  Victor S. Lempitsky,et al.  Fast ConvNets Using Group-Wise Brain Damage , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  R. Venkatesh Babu,et al.  Data-free Parameter Pruning for Deep Neural Networks , 2015, BMVC.

[21]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[22]  Yixin Chen,et al.  Compressing Neural Networks with the Hashing Trick , 2015, ICML.

[23]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[24]  Ming Yang,et al.  Compressing Deep Convolutional Networks using Vector Quantization , 2014, ArXiv.

[25]  Joan Bruna,et al.  Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation , 2014, NIPS.

[26]  Vincent Lepetit,et al.  Learning Separable Filters , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[28]  Enhong Chen,et al.  Image Denoising and Inpainting with Deep Neural Networks , 2012, NIPS.

[29]  Shiliang Sun,et al.  Multitask Twin Support Vector Machines , 2012, ICONIP.

[30]  Cewu Lu,et al.  Image smoothing via L0 gradient minimization , 2011, ACM Trans. Graph..

[31]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[32]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.