An energy-efficient stochastic computational deep belief network

Deep neural networks (DNNs) are effective machine learning models to solve a large class of recognition problems, including the classification of nonlinearly separable patterns. The applications of DNNs are, however, limited by the large size and high energy consumption of the networks. Recently, stochastic computation (SC) has been considered to implement DNNs to reduce the hardware cost. However, it requires a large number of random number generators (RNGs) that lower the energy efficiency of the network. To overcome these limitations, we propose the design of an energy-efficient deep belief network (DBN) based on stochastic computation. An approximate SC activation unit (A-SCAU) is designed to implement different types of activation functions in the neurons. The A-SCAU is immune to signal correlations, so the RNGs can be shared among all neurons in the same layer with no accuracy loss. The area and energy of the proposed design are 5.27% and 3.31% (or 26.55% and 29.89%) of a 32-bit floating-point (or an 8-bit fixed-point) implementation. It is shown that the proposed SC-DBN design achieves a higher classification accuracy compared to the fixed-point implementation. The accuracy is only lower by 0.12% than the floating-point design at a similar computation speed, but with a significantly lower energy consumption.

[1]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[3]  David J. Lilja,et al.  An FPGA implementation of a Restricted Boltzmann Machine classifier using stochastic bit streams , 2015, 2015 IEEE 26th International Conference on Application-specific Systems, Architectures and Processors (ASAP).

[4]  Naoya Onizawa,et al.  VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing , 2017, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[5]  John P. Hayes,et al.  Dimension reduction in statistical simulation of digital circuits , 2015, SpringSim.

[6]  John P. Hayes.  Introduction to stochastic computing and its challenges , 2015, 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[7]  Kiyoung Choi,et al.  Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[8]  Ji Li,et al.  DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks , 2016, 2016 IEEE 34th International Conference on Computer Design (ICCD).

[9]  John P. Hayes,et al.  Survey of Stochastic Computing , 2013, TECS.

[10]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[11]  Antoni Morro,et al.  Probabilistic-based neural network implementation , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[12]  Kia Bazargan,et al.  The synthesis of linear Finite State Machine-based Stochastic Computational Elements , 2012, 17th Asia and South Pacific Design Automation Conference.

[13]  Brian R. Gaines,et al.  Stochastic Computing Systems , 1969 .

[14]  Andreas G. Andreou,et al.  FPGA implementation of a Deep Belief Network architecture for character recognition using stochastic computation , 2015, 2015 49th Annual Conference on Information Sciences and Systems (CISS).

[15]  Dong Yu,et al.  Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition , 2012, IEEE Transactions on Audio, Speech, and Language Processing.

[16]  Feng Ran,et al.  A hardware implementation of a radial basis function neural network using stochastic logic , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[17]  Howard C. Card,et al.  Stochastic Neural Computation II: Soft Competitive Learning , 2001, IEEE Trans. Computers.

[18]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[19]  Howard C. Card,et al.  Stochastic Neural Computation I: Computational Elements , 2001, IEEE Trans. Computers.

[20]  Dumitru Erhan,et al.  Deep Neural Networks for Object Detection , 2013, NIPS.

[21]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.