Gradient Descent Using Stochastic Circuits for Efficient Training of Learning Machines

Gradient descent (GD) is a widely used optimization algorithm in machine learning. In this paper, a novel stochastic computing GD circuit (SC-GDC) is proposed by encoding the gradient information in stochastic sequences. Inspired by the structure of a neuron, a stochastic integrator is used to optimize the weights in a learning machine by its “inhibitory” and “excitatory” inputs. Specifically, two AND (or XNOR) gates for the unipolar representation (or the bipolar representation) and one stochastic integrator are, respectively, used to implement the multiplications and accumulations in a GD algorithm. Thus, the SC-GDC is very area- and power-efficient. As per the formulation of the proposed SC-GDC, it provides unbiased estimate of the optimized weights in a learning algorithm. The proposed SC-GDC is then used to implement a least-mean-square algorithm and a softmax regression. With a similar accuracy, the proposed design achieves more than <inline-formula> <tex-math notation="LaTeX">$30 \times $ </tex-math></inline-formula> improvement in throughput per area (TPA) and consumes less than 13% of the energy per training sample, compared with a fixed-point implementation. Moreover, a signed SC-GDC is proposed for training complex neural networks (NNs). It is shown that for a 784-128-128-10 fully connected NN, the signed SC-GDC produces a similar training result with its fixed-point counterpart, while achieving more than 90% energy saving and 82% reduction in training time with more than <inline-formula> <tex-math notation="LaTeX">$50 \times $ </tex-math></inline-formula> improvement in TPA.

[1]  Kia Bazargan,et al.  IIR filters using stochastic arithmetic , 2014, 2014 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[2]  David J. Lilja,et al.  An FPGA implementation of a Restricted Boltzmann Machine classifier using stochastic bit streams , 2015, 2015 IEEE 26th International Conference on Application-specific Systems, Architectures and Processors (ASAP).

[3]  Kiyoung Choi,et al.  Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks , 2016, 2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC).

[4]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, NIPS.

[5]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[6]  Fabrizio Lombardi,et al.  A Stochastic Computational Multi-Layer Perceptron with Backward Propagation , 2018, IEEE Transactions on Computers.

[7]  Qinru Qiu,et al.  SC-DCNN: Highly-Scalable Deep Convolutional Neural Network using Stochastic Computing , 2016, ASPLOS.

[8]  Ran El-Yaniv,et al.  Binarized Neural Networks , 2016, ArXiv.

[9]  Feng Ran,et al.  A hardware implementation of a radial basis function neural network using stochastic logic , 2015, 2015 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[10]  Arash Ardakani,et al.  Stochastic Computing Can Improve Upon Digital Spiking Neural Networks , 2016, 2016 IEEE International Workshop on Signal Processing Systems (SiPS).

[11]  David A. Patterson,et al.  In-datacenter performance analysis of a tensor processing unit , 2017, 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

[12]  John P. Hayes,et al.  Energy-efficient hybrid stochastic-binary neural networks for near-sensor computing , 2017, Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017.

[13]  Naoya Onizawa,et al.  VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing , 2017, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[14]  Gu-Yeon Wei,et al.  Minerva: Enabling Low-Power, Highly-Accurate Deep Neural Network Accelerators , 2016, 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA).

[15]  Leibo Liu,et al.  An efficient hardware design for cerebellar models using approximate circuits: special session paper , 2017, CODES+ISSS.

[16]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[17]  Bernard Brezzo,et al.  TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip , 2015, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[18]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[19]  Cong Xu,et al.  TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning , 2017, NIPS.

[20]  Ji Li,et al.  Softmax Regression Design for Stochastic Computing Based Deep Convolutional Neural Networks , 2017, ACM Great Lakes Symposium on VLSI.

[21]  Zhongfeng Wang,et al.  Fully-Parallel Area-Efficient Deep Neural Network Design Using Stochastic Computing , 2017, IEEE Transactions on Circuits and Systems II: Express Briefs.

[22]  Duncan G. Elliott,et al.  A High-Performance and Energy-Efficient FIR Adaptive Filter Using Approximate Distributed Arithmetic Circuits , 2019, IEEE Transactions on Circuits and Systems I: Regular Papers.

[23]  Chen Wang,et al.  An efficient implementation of numerical integration using logical computation on stochastic bit streams , 2012, 2012 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[24]  Howard C. Card,et al.  Stochastic Neural Computation I: Computational Elements , 2001, IEEE Trans. Computers.

[25]  B. Farhang-Boroujeny,et al.  Adaptive Filters: Theory and Applications , 1999 .

[26]  Jie Han,et al.  Hardware ODE solvers using stochastic circuits , 2017, 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC).

[27]  Jie Han,et al.  Toward Energy-Efficient Stochastic Circuits Using Parallel Sobol Sequences , 2018, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[28]  John P. Hayes,et al.  The Promise and Challenge of Stochastic Computing , 2018, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[29]  Fabrizio Lombardi,et al.  Adaptive Filter Design Using Stochastic Circuits , 2016, 2016 IEEE Computer Society Annual Symposium on VLSI (ISVLSI).

[30]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.