A Multi-objective Particle Swarm Optimization for Neural Networks Pruning

There is a ruling maxim in deep learning land, bigger is better. However, bigger neural network provides higher performance but also expensive computation, memory and energy. The simplified model which preserves the accuracy of original network arouses a growing interest. A simple yet efficient method is pruning, which cuts off unimportant synapses and neurons. Therefore, it is crucial to identify important parts from the given numerous connections. In this paper, we use the evolutionary pruning method to simplify the structure of deep neural networks. A multi-objective neural networks pruning model which balances the accuracy and the sparse ratio of networks is proposed and we solve this model with particle swarm optimization (PSO) method. Furthermore, we fine-tune the network which is obtained by pruning to obtain better pruning result. The framework of alternate pruning and fine-tuning operations is used to achieve more prominent pruning effect. In experimental studies, we prune LeNet on MNIST and shallow VGGNet on CIFAR-10. Experimental results demonstrate that our method could prune over 80% weights in general with no loss of accuracy.

[1]  Wonyong Sung,et al.  Structured Pruning of Deep Convolutional Neural Networks , 2015, ACM J. Emerg. Technol. Comput. Syst..

[2]  Xiaogang Wang,et al.  Structure Learning for Deep Neural Networks Based on Multiobjective Optimization , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[3]  M Reyes Sierra,et al.  Multi-Objective Particle Swarm Optimizers: A Survey of the State-of-the-Art , 2006 .

[4]  Xiangyu Zhang,et al.  Channel Pruning for Accelerating Very Deep Neural Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Qingfu Zhang,et al.  MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition , 2007, IEEE Transactions on Evolutionary Computation.

[6]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[7]  Misha Denil,et al.  Predicting Parameters in Deep Learning , 2014 .

[8]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Maoguo Gong,et al.  Decomposition-Based Evolutionary Multiobjective Optimization to Self-Paced Learning , 2019, IEEE Transactions on Evolutionary Computation.

[10]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[11]  Xin Yao,et al.  An Evolutionary Multiobjective Approach to Sparse Reconstruction , 2014, IEEE Transactions on Evolutionary Computation.

[12]  R. Venkatesh Babu,et al.  Data-free Parameter Pruning for Deep Neural Networks , 2015, BMVC.

[13]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[14]  Maoguo Gong,et al.  A Multiobjective Sparse Feature Learning Model for Deep Neural Networks , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[16]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[17]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[18]  Giovanna Castellano,et al.  An iterative pruning algorithm for feedforward neural networks , 1997, IEEE Trans. Neural Networks.

[19]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[20]  Maoguo Gong,et al.  Multiobjective Immune Algorithm with Nondominated Neighbor-Based Selection , 2008, Evolutionary Computation.

[21]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Mengjie Zhang,et al.  Evolving Deep Convolutional Neural Networks for Image Classification , 2017, IEEE Transactions on Evolutionary Computation.

[23]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[24]  Andy J. Keane,et al.  Pruning backpropagation neural networks using modern stochastic optimisation techniques , 1997, Neural Computing & Applications.

[25]  Maoguo Gong,et al.  A multi-objective memetic algorithm for low rank and sparse matrix decomposition , 2018, Inf. Sci..

[26]  Carlos A. Coello Coello,et al.  Handling multiple objectives with particle swarm optimization , 2004, IEEE Transactions on Evolutionary Computation.