Dither computing: a hybrid deterministic-stochastic computing framework

Stochastic computing has a long history as an alternative method of performing arithmetic on a computer. While it can be considered an unbiased estimator of real numbers, it has a variance and MSE on the order of $\displaystyle \Omega(\frac{1}{N})$. On the other hand, deterministic variants of stochastic computing remove the stochastic aspect, but cannot approximate arbitrary real numbers with arbitrary precision and are biased estimators. However, they have an asymptotically superior MSE on the order of $O(\frac{1}{N^{2}})$. Recent results in deep learning with stochastic rounding suggest that the bias in the rounding can degrade performance. We proposed an alternative framework, called dither computing, that combines aspects of stochastic computing and its deterministic variants and that can perform computing with similar efficiency, is unbiased, and with a variance and MSE also on the optimal order of $\displaystyle \Theta(\frac{1}{N^{2}})$. We also show that it can be beneficial in stochastic rounding applications as well. We provide implementation details and give experimental results to comparatively show the benefits of the proposed scheme.

[1]  V. Strassen.  Gaussian elimination is not optimal , 1969 .

[2]  Fabrizio Lombardi,et al.  A Survey of Stochastic Computing Neural Networks for Machine Learning Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[3]  Don Coppersmith,et al.  Matrix multiplication via arithmetic progressions , 1987, STOC.

[4]  Marc D. Riedel,et al.  A deterministic approach to stochastic computation , 2016, 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[5]  Nicu Sebe,et al.  Binary Neural Networks: A Survey , 2020, Pattern Recognit..

[6]  John P. Hayes,et al.  Survey of Stochastic Computing , 2013, TECS.

[7]  Steve B. Furber,et al.  Stochastic rounding and reduced-precision fixed-point arithmetic for solving neural ODEs , 2019, ArXiv.

[8]  Brian R. Gaines,et al.  Stochastic computing , 1967, AFIPS '67 (Spring).

[9]  Markus Höhfeld,et al.  Probabilistic rounding in neural network learning with limited precision , 1992, Neurocomputing.

[10]  John P. Hayes,et al.  Analyzing and controlling accuracy in stochastic circuits , 2014, 2014 IEEE 32nd International Conference on Computer Design (ICCD).

[11]  Mário P. Véstias,et al.  Enhancing stochastic computations via process variation , 2015, 2015 25th International Conference on Field Programmable Logic and Applications (FPL).

[12]  Martin Margala,et al.  Exploration of Low Numeric Precision Deep Learning Inference Using Intel® FPGAs , 2018, 2018 IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM).

[13]  Swagath Venkataramani,et al.  Accurate and Efficient 2-bit Quantized Neural Networks , 2019, MLSys.

[14]  Josh Alman,et al.  A Refined Laser Method and Faster Matrix Multiplication , 2020, SODA.

[15]  Elaine J. Weyuker,et al.  Computability, complexity, and languages - fundamentals of theoretical computer science , 2014, Computer science and applied mathematics.

[16]  Pritish Narayanan,et al.  Deep Learning with Limited Numerical Precision , 2015, ICML.

[17]  Nicholas J. Higham,et al.  Stochastic Rounding and Its Probabilistic Backward Error Analysis , 2020, SIAM J. Sci. Comput..

[18]  Tayfun Gokmen,et al.  Training Deep Convolutional Neural Networks with Resistive Cross-Point Devices , 2017, Front. Neurosci..

[19]  Roland Vollgraf,et al.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms , 2017, ArXiv.