Towards acceleration of deep convolutional neural networks using stochastic computing

In recent years, Deep Convolutional Neural Network (DCNN) has become the dominant approach for almost all recognition and detection tasks and outperformed humans on certain tasks. Nevertheless, the high power consumptions and complex topologies have hindered the widespread deployment of DCNNs, particularly in wearable devices and embedded systems with limited area and power budget. This paper presents a fully parallel and scalable hardware-based DCNN design using Stochastic Computing (SC), which leverages the energy-accuracy trade-off through optimizing SC components in different layers. We first conduct a detailed investigation of the Approximate Parallel Counter (APC) based neuron and multiplexer-based neuron using SC, and analyze the impacts of various design parameters, such as bit stream length and input number, on the energy/power/area/accuracy of the neuron cell. Then, from an architecture perspective, the influence of inaccuracy of neurons in different layers on the overall DCNN accuracy (i.e., software accuracy of the entire DCNN) is studied. Accordingly, a structure optimization method is proposed for a general DCNN architecture, in which neurons in different layers are implemented with optimized SC components, so as to reduce the area, power, and energy of the DCNN while maintaining the overall network performance in terms of accuracy. Experimental results show that the proposed approach can find a satisfactory DCNN configuration, which achieves 55X, 151X, and 2X improvement in terms of area, power and energy, respectively, while the error is increased by 2.86%, compared with the conventional binary ASIC implementation.

[1]  Pai-Shun Ting,et al.  Stochastic Logic Realization of Matrix Operations , 2014, 2014 17th Euromicro Conference on Digital System Design.

[2]  Kia Bazargan,et al.  The synthesis of complex arithmetic computation on stochastic bit streams using sequential logic , 2012, 2012 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[3]  Ji Li,et al.  Accelerating soft-error-rate (SER) estimation in the presence of single event transients , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[4]  Kiyoung Choi,et al.  Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[5]  Ji Li,et al.  DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks , 2016, 2016 IEEE 34th International Conference on Computer Design (ICCD).

[6]  Andreas G. Andreou,et al.  FPGA implementation of a Deep Belief Network architecture for character recognition using stochastic computation , 2015, 2015 49th Annual Conference on Information Sciences and Systems (CISS).

[7]  Qinru Qiu,et al.  Designing reconfigurable large-scale deep learning systems using stochastic computing , 2016, 2016 IEEE International Conference on Rebooting Computing (ICRC).

[8]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[9]  Howard C. Card,et al.  Stochastic Neural Computation I: Computational Elements , 2001, IEEE Trans. Computers.

[10]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[12]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[13]  Feng Ran,et al.  A hardware implementation of a radial basis function neural network using stochastic logic , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[14]  Howard C. Card,et al.  Stochastic Neural Computation II: Soft Competitive Learning , 2001, IEEE Trans. Computers.

[15]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[16]  Ji Li,et al.  Joint Soft-Error-Rate (SER) Estimation for Combinational Logic and Sequential Elements , 2016, 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[17]  Soheil Ghiasi,et al.  Design space exploration of FPGA-based Deep Convolutional Neural Networks , 2016, 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC).

[18]  Kiyoung Choi,et al.  Efficient FPGA acceleration of Convolutional Neural Networks using logical-3D compute array , 2016, 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[19]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Tara N. Sainath,et al.  Deep convolutional neural networks for LVCSR , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[21]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.