Analysis of quantization effects on high-order function neural networks

Abstract In this paper we investigate the combined effects of quantization and clipping on high-order function neural networks (HOFNN). Statistical models are used to analyze the effects of quantization in a digital implementation. We analyze the performance degradation caused as a function of the number of fixed-point and floating-point quantization bits under the assumption of different probability distributions for the quantized variables, and then compare the training performance between situations with and without weight clipping. We establish and analyze the relationships for a true nonlinear neuron between inputs and outputs bit resolution, training and quantization methods, network order and performance degradation, all based on statistical models, and for on-chip and off-chip training. Our experimental simulation results verify the presented theoretical analysis.

[1]  Bo Zhang,et al.  Fast Learning Algorithms for Feedforward Neural Networks , 2004, Applied Intelligence.

[2]  Howard C. Card,et al.  Tolerance to analog hardware of on-chip learning in backpropagation networks , 1995, IEEE Trans. Neural Networks.

[3]  F. Kashefi.  Rapidly training device for fiber-optic neural network , 1999 .

[4]  Tetsuro Itakura,et al.  Neuro chips with on-chip back-propagation and/or Hebbian learning , 1992 .

[5]  Yun Xie,et al.  Analysis of the effects of quantization in multilayer neural networks using a statistical model , 1992, IEEE Trans. Neural Networks.

[6]  Ronald S. Gyurcsik,et al.  Toward a general-purpose analog VLSI neural network with on-chip learning , 1997, IEEE Trans. Neural Networks.

[7]  Hiroshi Yamamoto,et al.  Reduction of required precision bits for back-propagation applied to pattern recognition , 1993, IEEE Trans. Neural Networks.

[8]  Georges G. E. Gielen,et al.  Backpropagation Analysis of the Limited Precision on High-Order Function Neural Networks , 2004, ISNN.

[9]  Leonardo Maria Reyneri,et al.  An Analysis on the Performance of Silicon Implementations of Backpropagation Algorithms for Artificial Neural Networks , 1991, IEEE Trans. Computers.

[10]  Hon Keung Kwan,et al.  Multilayer feedforward neural networks with single powers-of-two weights , 1993, IEEE Trans. Signal Process..

[11]  Jerzy B. Lont.  Analog CMOS implementation of a multi-layer perceptron with nonlinear synapses , 1992, IEEE Trans. Neural Networks.

[12]  Georges Gielen,et al.  The effects of quantization on high order function neural networks , 2001, Neural Networks for Signal Processing XI: Proceedings of the 2001 IEEE Signal Processing Society Workshop (IEEE Cat. No.01TH8584).

[13]  Sina Balkir,et al.  ANNSyS (an analog neural network synthesis system) , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[14]  Georges G. E. Gielen,et al.  A fast learning algorithm for time-delay neural networks , 2002, Inf. Sci..

[15]  Anastasios N. Venetsanopoulos,et al.  Fast learning algorithms for neural networks , 1992 .

[16]  Emile Fiesler,et al.  Neural Network Adaptations to Hardware Implementations , 1997 .

[17]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[18]  Russell Beale,et al.  Handbook of Neural Computation , 1996 .

[19]  Georges G. E. Gielen,et al.  The Effects of Quantization on Multi-Layer Feedforward Neural Networks , 2003, Int. J. Pattern Recognit. Artif. Intell..

[20]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[21]  Günhan Dündar,et al.  The effects of quantization on multilayer neural networks , 1995, IEEE Trans. Neural Networks.

[22]  J. J. Paulos,et al.  Artificial neural networks using MOS analog multipliers , 1990 .

[23]  Marco Gori,et al.  Optimal convergence of on-line backpropagation , 1996, IEEE Trans. Neural Networks.

[24]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[25]  Kishan G. Mehrotra,et al.  Efficient classification for multiclass problems using modular neural networks , 1995, IEEE Trans. Neural Networks.

[26]  Stephen A. Dyer,et al.  Digital signal processing , 2018, 8th International Multitopic Conference, 2004. Proceedings of INMIC 2004..