Accuracy Evaluation of Deep belief Networks with Fixed-Point Arithmetic

Deep Belief Networks (DBNs) are state-of-art Machine Learning techniques and one of the most important unsupervised learning algorithms. Training DBNs is computationally intensive which naturally leads to investigate FPGA acceleration. Fixed-point arithmetic can be used when implementing DBNs in FPGAs to reduce execution time, but it is not clear the implications for accuracy. Previous studies have focused only on accelerators using some fixed bit-widths. A contribution of this paper is to demonstrate the bit-width effect on various configurations of DBNs in a comprehensive way by experimental evaluation. Explicit performance changing points are found using various bit-widths. The impact of sigmoid function approximation, required part of DBNs, is evaluated. A solution of mixed bit-widths DBN is proposed, fitting the bit-widths of FPGA primitives and gaining similar performance to the software implementation. Our results provide a guide to inform the design choices on bit-widths when implementing DBNs in FPGAs documenting clearly the trade-off in accuracy.

[1]  Paul Chow,et al.  A multi-FPGA architecture for stochastic Restricted Boltzmann Machines , 2009, 2009 International Conference on Field Programmable Logic and Applications.

[2]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[3]  Paul Chow,et al.  Building a multi-FPGA virtualized restricted boltzmann machine architecture using embedded MPI , 2011, FPGA '11.

[4]  M.A.A. Arroyo Leon,et al.  An artificial neural network on a field programmable gate array as a virtual sensor , 1999 .

[5]  Sorin Draghici,et al.  On the capabilities of neural networks using limited precision weights , 2002, Neural Networks.

[6]  C. Alippi,et al.  Simple approximation of sigmoidal functions: realistic design of digital neural networks capable of learning , 1991, 1991., IEEE International Sympoisum on Circuits and Systems.

[7]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[8]  Honglak Lee,et al.  Sparse deep belief net model for visual area V2 , 2007, NIPS.

[9]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[10]  DraghiciSorin.  On the capabilities of neural networks using limited precision weights , 2002 .

[11]  Medhat A. Moussa,et al.  Resource Efficient Arithmetic Effects on RBM Neural Network Solution Quality Using MNIST , 2011, 2011 International Conference on Reconfigurable Computing and FPGAs.

[12]  K. M. Curtis,et al.  Piecewise linear approximation applied to nonlinear function of a neural network , 1997 .

[13]  H.S. Abdel-Aty-Zohdy,et al.  Implementation of programmable digital sigmoid function circuit for neuro-computing , 1998, 1998 Midwest Symposium on Circuits and Systems (Cat. No. 98CB36268).

[14]  Kunle Olukotun,et al.  A Large-Scale Architecture for Restricted Boltzmann Machines , 2010, 2010 18th IEEE Annual International Symposium on Field-Programmable Custom Computing Machines.

[15]  Marc'Aurelio Ranzato,et al.  Building high-level features using large scale unsupervised learning , 2011, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.