BitFlow-Net: Toward Fully Binarized Convolutional Neural Networks

Binarization can greatly compress and accelerate deep convolutional neural networks (CNNs) for real-time industrial applications. However, existing binarized CNNs (BCNNs) rely on scaling factor (SF) and batch normalization (BatchNorm) that still involve resource-consuming floating-point multiplication operations. Addressing the limitation, an improved BCNN named BitFlow-Net is proposed, which replaces floating-point operations with integer addition in middle layers. First, it is derived that the SF is only effective in back-propagation process, whereas it is counteracted by BatchNorm in inference process. Then, in model running phase, the SF and BatchNorm are fused into an integer addition, named BatchShift. Consequently, the data flow in middle layers is fully binarized during modeling running phase. To verify its potential in industrial applications with multiclass and binary classification tasks, the BitFlow-Net is built based on AlexNet and verified on two large image datasets, i.e., ImageNet and 11K Hands. Experimental results show that the BitFlow-Net can remove all floating-point operations in middle layers of BCNNs and greatly reduce the memory for both cases without affecting the accuracy. Particularly, the BitFlow-Net can achieve the accuracy comparable to that of the full-precision AlexNet network in the binary classification task.

[1]  Jingchang Huang,et al.  Design of an Acoustic Target Classification System Based on Small-Aperture Microphone Array , 2015, IEEE Transactions on Instrumentation and Measurement.

[2]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[3]  Viktor K. Prasanna,et al.  Analysis of high-performance floating-point arithmetic on FPGAs , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[4]  Zhenbao Liu,et al.  Vehicle Detection in Aerial Images Using Rotation-Invariant Cascaded Forest , 2019, IEEE Access.

[5]  Yoshua Bengio,et al.  BinaryConnect: Training Deep Neural Networks with binary weights during propagations , 2015, NIPS.

[6]  Carlo Alberto Avizzano,et al.  A Smart Monitoring System for Automatic Welding Defect Detection , 2019, IEEE Transactions on Industrial Electronics.

[7]  Di Guo,et al.  Accelerated Nuclear Magnetic Resonance Spectroscopy with Deep Learning , 2019, Angewandte Chemie.

[8]  Niraj K. Jha,et al.  Grow and Prune Compact, Fast, and Accurate LSTMs , 2018, IEEE Transactions on Computers.

[9]  Peijie Lin,et al.  Deep residual network based fault detection and diagnosis of photovoltaic arrays using current-voltage curves and ambient conditions , 2019, Energy Conversion and Management.

[10]  Mark D. McDonnell,et al.  Training wide residual networks for deployment using a single bit for each weight , 2018, ICLR.

[11]  Oral Büyüköztürk,et al.  Deep Learning‐Based Crack Damage Detection Using Convolutional Neural Networks , 2017, Comput. Aided Civ. Infrastructure Eng..

[12]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[13]  Dong-Joong Kang,et al.  Machine learning-based imaging system for surface defect inspection , 2016, International Journal of Precision Engineering and Manufacturing-Green Technology.

[14]  Yongliang Wang,et al.  A 34-FPS 698-GOP/s/W Binarized Deep Neural Network-Based Natural Scene Text Interpretation Accelerator for Mobile Edge Computing , 2019, IEEE Transactions on Industrial Electronics.

[15]  Shuchang Zhou,et al.  DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients , 2016, ArXiv.

[16]  Rongrong Ji,et al.  Holistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Qixiang Ye,et al.  A scalable convolutional neural network for task-specified scenarios via knowledge distillation , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[19]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[20]  Mianxiong Dong,et al.  Deep Learning for Smart Industry: Efficient Manufacture Inspection System With Fog Computing , 2018, IEEE Transactions on Industrial Informatics.

[21]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, ArXiv.

[22]  Bin Wang,et al.  Where to Prune: Using LSTM to Guide End-to-end Pruning , 2018, IJCAI.

[23]  Lei Liu,et al.  Fast CNN Pruning via Redundancy-Aware Training , 2018, ICANN.

[24]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[25]  Wei Liu,et al.  Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm , 2018, ECCV.

[26]  Song Han,et al.  Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding , 2015, ICLR.

[27]  Mahmoud Afifi,et al.  11K Hands: Gender recognition and biometric identification using a large dataset of hand images , 2017, Multimedia Tools and Applications.

[28]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[29]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[30]  Ming Yang,et al.  Compressing Deep Convolutional Networks using Vector Quantization , 2014, ArXiv.

[31]  Jingchang Huang,et al.  A Two-Stage Detection Method for Moving Targets in the Wild Based on Microphone Array , 2015, IEEE Sensors Journal.

[32]  Takehisa Yairi,et al.  A review on the application of deep learning in system health management , 2018, Mechanical Systems and Signal Processing.

[33]  Shuang Wu,et al.  Training and Inference with Integers in Deep Neural Networks , 2018, ICLR.

[34]  Igor Carron,et al.  XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks , 2016 .

[35]  Jiheon Kang,et al.  Novel Leakage Detection by Ensemble CNN-SVM and Graph-Based Localization in Water Distribution Systems , 2018, IEEE Transactions on Industrial Electronics.

[36]  Liang Gao,et al.  A New Convolutional Neural Network-Based Data-Driven Fault Diagnosis Method , 2018, IEEE Transactions on Industrial Electronics.

[37]  Gang Hua,et al.  How to Train a Compact Binary Neural Network with High Accuracy? , 2017, AAAI.

[38]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Giovanni De Magistris,et al.  Spatio-Temporal Anomaly Detection for Industrial Robots through Prediction in Unsupervised Feature Space , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[40]  Lin Xu,et al.  Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights , 2017, ICLR.