Optoelectronic analogs of self-programming neural nets: architecture and methodologies for implementing fast stochastic learning by simulated annealing.

Self-organization and learning is a distinctive feature of neural nets and processors that sets them apart from conventional approaches to signal processing. It leads to self-programmability which alleviates the problem of programming complexity in artificial neural nets. In this paper architectures for partitioning an optoelectronic analog of a neural net into distinct layers with prescribed interconnectivity pattern to enable stochastic learning by simulated annealing in the context of a Boltzmann machine are presented. Stochastic learning is of interest because of its relevance to the role of noise in biological neural nets. Practical considerations and methodologies for appreciably accelerating stochastic learning in such a multilayered net are described. These include the use of parallel optical computing of the global energy of the net, the use of fast nonvolatile programmable spatial light modulators to realize fast plasticity, optical generation of random number arrays, and an adaptive noisy thresholding scheme that also makes stochastic learning more biologically plausible. The findings reported predict optoelectronic chips that can be used in the realization of optical learning machines.

[1]  P Chavel,et al.  Optical generation of random-number arrays for on-chip massively parallel Monte Carlo cellular processors. , 1987, Optics letters.

[2]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[3]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[4]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[5]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[6]  Richard M. Karp,et al.  Combinatorics, complexity, and randomness , 1986, CACM.

[7]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[8]  G. M. Morris,et al.  Generation of random arrays using clipped laser speckle. , 1986, Applied optics.

[9]  L. Marton,et al.  Advances in Electronics and Electron Physics , 1958 .

[10]  C. Lee Giles,et al.  Optical Adaptive Associative Computer Architectures , 1985, COMPCON.

[11]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[12]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[13]  J. Cowan,et al.  Large Scale Spatially Organized Activity in Neural Nets , 1980 .

[14]  Demetri Psaltis,et al.  Acoustooptic Implementation Of Neural Network Models , 1985 .

[15]  D Psaltis,et al.  Optical information processing based on an associative-memory model of neural nets with thresholding and feedback. , 1985, Optics letters.

[16]  T. Kurono,et al.  Photon-counting imaging and its application , 1986 .

[17]  D B Bounds.  Numerical simulations of Boltzmann Machines , 1987 .

[18]  B H Soffer,et al.  Associative holographic memory with feedback using phase-conjugate mirrors. , 1986, Optics letters.

[19]  N H Farhat.  Architectures for optoelectronic analogs of self-organizing neural networks. , 1987, Optics letters.

[20]  D Psaltis,et al.  Optical implementation of the Hopfield model. , 1985, Applied optics.

[21]  Yehoshua Y. Zeevi,et al.  Special issue on neural and sensory information processing , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[22]  J. Goodman,et al.  Neural networks for computation: number representations and programming complexity. , 1986, Applied optics.

[23]  Demetri Psaltis,et al.  New Approach To Optical Information-Processing Based On The Hopfield Model , 1984 .

[25]  Nabil H. Farhat,et al.  2.3 – Optical Implementation of Associative Memory Based on Models of Neural Networks , 1987 .

[26]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[27]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[28]  D. Willshaw.  A simple network capable of inductive generalization , 1972, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[29]  Demetri Psaltis,et al.  Two-Dimensional Magneto-Optic Spatial Light Modulator For Signal Processing , 1983 .

[30]  D. Z. Anderson,et al.  Coherent optical eigenstate memory. , 1986, Optics letters.

[31]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..

[32]  A. Yariv,et al.  Associative memories based on message-bearing optical modes in phase-conjugate resonators. , 1986, Optics letters.

[33]  G. M. Morris.  Optical Computing By Monte Carlo Methods , 1984 .

[34]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.