Learning in Stochastic Bit Stream Neural Networks

This paper presents learning techniques for a novel feedforward stochastic neural network. The model uses stochastic weights and the "bit stream" data representation. It has a clean analysable functionality and is very attractive with its great potential to be implemented in hardware using standard digital VLSI technology. The design allows simulation at three different levels and learning techniques are described for each level. The lowest level corresponds to on-chip learning. Simulation results on three benchmark MONK's problems and handwritten digit recognition with a clean set of 500 16 x 16 pixel digits demonstrate that the new model is powerful enough for the real world applications. Copyright 1996 Elsevier Science Ltd

[1]  John E. Moody,et al.  Towards Faster Stochastic Gradient Search , 1991, NIPS.

[2]  Yasuji Sawada,et al.  Functional abilities of a stochastic logic neural network , 1992, IEEE Trans. Neural Networks.

[3]  John Shawe-Taylor,et al.  Probabilistic Bit Stream Neural Chip: Implementation , 1991 .

[4]  John Shawe-Taylor,et al.  A recurrent network with stochastic weights , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[5]  John Shawe-Taylor,et al.  Probabilistic Bit Stream Neural Chip: Theory , 1991 .

[6]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[7]  Paul J. Werbos,et al.  The roots of backpropagation , 1994 .

[8]  Kimmo Kaski,et al.  Pulse-density modulation technique in VLSI implementations of neural network algorithms , 1990 .

[9]  John Shawe-Taylor,et al.  Generating binary sequences for stochastic computing , 1994, IEEE Trans. Inf. Theory.

[10]  Alan F. Murray,et al.  Analogue Neural Vlsi: A Pulse Stream Approach , 1994 .

[11]  Brian R. Gaines,et al.  Stochastic Computing Systems , 1969 .

[12]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[13]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[14]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[15]  J. Tou.  Advances in Information Systems Science , 1970, Springer US.

[16]  J. Shawe-Taylor,et al.  A stochastic neural architecture that exploits dynamically reconfigurable FPGAs , 1993, [1993] Proceedings IEEE Workshop on FPGAs for Custom Computing Machines.