Asymmetric Ternary Networks

Deep Neural Networks (DNNs) are widely used in a variety of machine learning tasks currently, especially in speech recognition and image classification. However, the huge demand for memory and computational power makes DNNs cannot be deployed on embedded devices efficiently. In this paper, we propose asymmetric ternary networks (ATNs) – neural networks with weights constrained to ternary values (-α1,0,+α2), which can reduce the DNN models size by about 16 × compared with 32-bits full precision models. Scaling factors {α1,α2} are used to reduce the quantization loss between ternary weights and full precision weights. We compare ATNs with recently proposed ternary weight networks (TWNs) and full precision networks on CIFAR-10 and ImageNet datasets. The results show that our ATN models outperform full precision models of VGG13, VGG16 by 0.11%, 0.33% respectively on CIFAR-10. On ImageNet, our model outperforms TWN AlexNet model by 2.25% of Top-1 accuracy and has only 0.63% accuracy degradation over the fullprecision counterpart.

[1]  Bin Liu,et al.  Ternary Weight Networks , 2016, ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  Yoshua Bengio,et al.  Training deep neural networks with low precision multiplications , 2014 .

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, ArXiv.

[6]  Vincent Vanhoucke,et al.  Improving the speed of neural networks on CPUs , 2011 .

[7]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[8]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[9]  Kyuyeon Hwang,et al.  Fixed-point feedforward deep neural network design using weights +1, 0, and −1 , 2014, 2014 IEEE Workshop on Signal Processing Systems (SiPS).

[10]  Yixin Chen,et al.  Compressing Neural Networks with the Hashing Trick , 2015, ICML.

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[12]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[13]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[14]  Igor Carron,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016 .

[15]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Yoshua Bengio,et al.  Neural Networks with Few Multiplications , 2015, ICLR.

[17]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Yoshua Bengio,et al.  Scaling learning algorithms towards AI , 2007 .

[19]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Ming Yang,et al.  Compressing Deep Convolutional Networks using Vector Quantization , 2014, ArXiv.