Linear Computation Coding

We introduce the new concept of computation coding. For linear functions, we present an algorithm to reduce the computational cost of multiplying an arbitrary given matrix with an unknown vector. It decomposes the given matrix into the product of codebook and wiring matrices whose entries are either zero or signed integer powers of two.For a typical implementation of deep neural networks, the proposed algorithm reduces the number of required addition units several times. To achieve the accuracy of 16-bit signed integer arithmetic for 4k-vectors, no multipliers and only 1.5 adders per matrix entry are needed.

[1]  Ryan Williams,et al.  Matrix-vector multiplication in sub-quadratic time: (some preprocessing required) , 2007, SODA '07.

[2]  Lingamneni Avinash,et al.  Ten Years of Building Broken Chips: The Physics and Engineering of Inexact Computing , 2013, TECS.

[3]  Arnold Schönhage,et al.  Schnelle Multiplikation großer Zahlen , 1971, Computing.

[4]  V. Lefèvre,et al.  Moyens arithmetiques pour un calcul fiable , 2000 .

[5]  Ralf R. Müller,et al.  On Approximation, Bounding & Exact Calculation of Average Block Error Probability for Random Code Ensembles , 2021, IEEE Transactions on Communications.

[6]  Nicolas Boullis,et al.  Some optimizations of hardware multiplication by constant matrices , 2005, IEEE Transactions on Computers.

[7]  David L. Neuhoff,et al.  Quantization , 2022, IEEE Trans. Inf. Theory.

[8]  Graham A. Jullien,et al.  Theory and applications for a double-base number system , 1997, Proceedings 13th IEEE Sympsoium on Computer Arithmetic.

[9]  Vassil S. Dimitrov,et al.  Area-Efficient Multipliers Based on Multiple-Radix Representations , 2011, IEEE Transactions on Computers.

[10]  T. Cover,et al.  Rate Distortion Theory , 2001 .

[11]  Mark G. Arnold,et al.  Redundant Logarithmic Arithmetic , 1990, IEEE Trans. Computers.

[12]  Martin Fürer.  Faster integer multiplication , 2007, STOC '07.

[13]  Andrew D. Booth,et al.  A SIGNED BINARY MULTIPLICATION TECHNIQUE , 1951 .

[14]  R. Hartley.  Subexpression sharing in filters using canonic signed digit multipliers , 1996 .

[15]  Don Coppersmith,et al.  Matrix multiplication via arithmetic progressions , 1987, STOC.

[16]  Jalil Boukhobza,et al.  Emerging Non-volatile Memories , 2017 .

[17]  Ali Bereyhi,et al.  Efficient Matrix Multiplication: The Sparse Power-of-2 Factorization , 2020, 2020 Information Theory and Applications Workshop (ITA).

[18]  M Itoh,et al.  Modified signed-digit arithmetic based on redundant bit representation. , 1994, Applied optics.

[19]  Max Welling,et al.  Bayesian Compression for Deep Learning , 2017, NIPS.

[20]  V. Strassen.  Gaussian elimination is not optimal , 1969 .

[21]  K. B. Oldham,et al.  An Atlas of Functions. , 1988 .

[22]  Song Han,et al.  Learning both Weights and Connections for Efficient Neural Network , 2015, NIPS.

[23]  Joris van der Hoeven,et al.  Even faster integer multiplication , 2014, J. Complex..

[24]  R. DeVore,et al.  Nonlinear approximation , 1998, Acta Numerica.

[25]  Tiefeng Jiang,et al.  The asymptotic distributions of the largest entries of sample correlation matrices , 2004, math/0406184.

[26]  Neri Merhav,et al.  Statistical Physics and Information Theory , 2010, Found. Trends Commun. Inf. Theory.

[27]  Anatolij A. Karatsuba,et al.  Multiplication of Multidigit Numbers on Automata , 1963 .

[28]  S. Cook,et al.  ON THE MINIMUM COMPUTATION TIME OF FUNCTIONS , 1969 .

[29]  M. Kao,et al.  3-D Stacked Technology of DRAM-Logic Controller Using Through-Silicon Via (TSV) , 2018, IEEE Journal of the Electron Devices Society.

[30]  Tom Goldstein,et al.  Finite-Alphabet MMSE Equalization for All-Digital Massive MU-MIMO mmWave Communication , 2020, IEEE Journal on Selected Areas in Communications.

[31]  Levent Aksoy,et al.  A novel method for the approximation of multiplierless constant matrix vector multiplication , 2015, 2015 IEEE 13th International Conference on Embedded and Ubiquitous Computing.

[32]  Edo Liberty,et al.  The Mailman algorithm: A note on matrix-vector multiplication , 2009, Inf. Process. Lett..

[33]  Erich Elsen,et al.  The Difficulty of Training Sparse Neural Networks , 2019, ArXiv.