Neuromorphic Synapses for Artificial Dendrites

We describe neuromorphic, variable-weight synapses onartificial dendrites that facilitate experimentation with correlativeadaptation rules. Attention is focused on those aspects of biologicalsynaptic function that could affect a neuromorphic network'scomputational power and adaptive capability. These include sublinearsummation, quantal synaptic noise, and independent adaptationof adjacent synapses. The diffusive nature of artificial dendritesadds considerable flexibility to the design of fast synapsesby allowing conductances to be enabled for short or for variablelengths of time. We present two complementary synapse designs:the shared conductance array and the self-timed synapse. Bothsynapse circuits behave as conductances to mimic biological synapsesand thus enable sublinear summation. The former achieves weightvariation by selecting different conductances from an on-chiparray, and the latter by modulating the length of time a constantconductance remains activated. Both work with our interchip communicationsystem, virtual wires. For the present purpose of comparing variousadaptation mechanisms in software, synaptic weights are storedoff chip. This simplifies the addition of quantal weight noiseand allows connections from different sources to the same dendriticcompartment to have independent weights.

[1]  C Koch,et al.  The biophysical properties of spines as a basis for their electrical function: a comment on Kawato & Tsukahara (1983). , 1985, Journal of theoretical biology.

[2]  Timothy H. Murphy,et al.  Ca2+ Imaging of CNS Axons in Culture Indicates Reliable Coupling between Single Action Potentials and Distal Functional Release Sites , 1996, Neuron.

[3]  W. Rall.  Distinguishing theoretical synaptic potentials computed for different soma-dendritic distributions of synaptic input. , 1967, Journal of neurophysiology.

[4]  Mona E. Zaghloul,et al.  An enhancement-mode MOS voltage-controlled linear resistor with large dynamic range , 1990 .

[5]  Tetsuro Itakura,et al.  Neuro chips with on-chip back-propagation and/or Hebbian learning , 1992 .

[6]  G. Major,et al.  The modelling of pyramidal neurones in the visual cortex , 1989 .

[7]  Roberto Malinow,et al.  Measuring the impact of probabilistic transmission on neuronal output , 1993, Neuron.

[8]  W. Crill,et al.  Specific membrane properties of cat motoneurones , 1974, The Journal of physiology.

[9]  Mona E. Zaghloul,et al.  VLSI implementation of synaptic weighting and summing in pulse coded neural-type cells , 1992, IEEE Trans. Neural Networks.

[10]  John G. Elias,et al.  Artificial Dendritic Trees , 1993, Neural Computation.

[11]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[12]  K. Stratford,et al.  Presynaptic release probability influences the locus of long-term potentiation , 1992, Nature.

[13]  Tobias Delbriick.  Silicon Retina with Correlation-Based, Velocitv-Tuned Pixels , 1993 .

[14]  John M. Bekkers,et al.  Quantal analysis of synaptic transmission in the central nervous system , 1994, Current Opinion in Neurobiology.

[15]  E. W. Kairiss,et al.  Hebbian synapses: biophysical mechanisms and algorithms. , 1990, Annual review of neuroscience.

[16]  Alan F. Murray,et al.  Analogue Neural Vlsi: A Pulse Stream Approach , 1994 .

[17]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[18]  David P. M. Northmore,et al.  Spike Train Processing by a Silicon Neuromorph: The Role of Sublinear Summation in Dendrites , 1996, Neural Computation.

[19]  M. S. Ghausi,et al.  Fully integrated active-RC filters using MOS and non-balanced structure , 1987 .

[20]  R S Zucker,et al.  Postsynaptic calcium is sufficient for potentiation of hippocampal synaptic transmission. , 1988, Science.

[21]  T. Bliss,et al.  Memories of NMDA receptors and LTP , 1995, Trends in Neurosciences.

[22]  Carver Mead,et al.  Analog VLSI and neural systems , 1989 .

[23]  R. Nicoll,et al.  Long-term potentiation is associated with increases in quantal content and quantal amplitude , 1992, Nature.

[24]  Yannis Tsividis,et al.  Floating voltage-controlled resistors in CMOS technology , 1982 .

[25]  Segal Sj.  NORPLANT contraceptive implants advancing. , 1984 .

[26]  E. R. Kandel,et al.  Synaptic transmission: A bidirectional and self-modifiable form of cell-cell communication , 1993, Cell.

[27]  Jouni Tomberg.  Synchronous Pulse Density Modulation in Neural Network Implementation , 1994 .

[28]  D. Blackman,et al.  A general purpose analog neural computer , 1989, International 1989 Joint Conference on Neural Networks.

[29]  David P. M. Northmore,et al.  An Analog Memory Circuit for Spiking Silicon Neurons , 1997, Neural Computation.

[30]  Andreas G. Andreou,et al.  A Contrast Sensitive Silicon Retina with Reciprocal Synapses , 1991, NIPS.

[31]  T. Bliss,et al.  A synaptic model of memory: long-term potentiation in the hippocampus , 1993, Nature.

[32]  John Lazzaro,et al.  Circuit Models of Sensory Transduction in the Cochlea , 1989, Analog VLSI Implementation of Neural Systems.

[33]  C. Stevens,et al.  Changes in reliability of synaptic function as a mechanism for plasticity , 1994, Nature.

[34]  G. Shepherd.  The Synaptic Organization of the Brain , 1979 .

[35]  Richard F. Lyon,et al.  Improved implementation of the silicon cochlea , 1992 .

[36]  Eric A. Vittoz,et al.  A communication architecture tailored for analog VLSI artificial neural networks: intrinsic performance and limitations , 1994, IEEE Trans. Neural Networks.

[37]  William A. Phillips,et al.  A Biologically Supported Error-Correcting Learning Rule , 1991, Neural Computation.

[38]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[39]  J. Clements,et al.  Cable properties of cat spinal motoneurones measured by combining voltage clamp, current clamp and intracellular staining. , 1989, The Journal of physiology.

[40]  T. H. Brown,et al.  Biophysical model of a Hebbian synapse. , 1990, Proceedings of the National Academy of Sciences of the United States of America.

[41]  A. Thakoor,et al.  Design of parallel hardware neural network systems from custom analog VLSI 'building block' chips , 1989, International 1989 Joint Conference on Neural Networks.

[42]  Nicholas T. Carnevale,et al.  Hebbian learning is jointly controlled by electrotonic and input structure , 1994 .

[43]  R. Malinow,et al.  The probability of transmitter release at a mammalian central synapse , 1993, Nature.

[44]  R. Tsien,et al.  Presynaptic component of long-term potentiation visualized at individual hippocampal synapses. , 1995, Science.

[45]  Yuzo Hirai.  Recent VLSI neural networks in Japan , 1993, J. VLSI Signal Process..

[46]  Alan F. Murray,et al.  Use of a-Si:H memory devices for non-volatile weight storage in artificial neural networks , 1993 .

[47]  R. Malinow,et al.  Activation of postsynaptically silent synapses during pairing-induced LTP in CA1 region of hippocampal slice , 1995, Nature.

[48]  B Sakmann,et al.  Detailed passive cable models of whole-cell recorded CA3 pyramidal neurons in rat hippocampal slices , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[49]  Bing J. Sheu,et al.  General-purpose neural chips with electrically programmable synapses and gain-adjustable neurons , 1992 .

[50]  James L. McClelland,et al.  James L. McClelland, David Rumelhart and the PDP Research Group, Parallel distributed processing: explorations in the microstructure of cognition . Vol. 1. Foundations . Vol. 2. Psychological and biological models . Cambridge MA: M.I.T. Press, 1987. , 1989, Journal of Child Language.

[51]  Mona E. Zaghloul,et al.  Silicon Implementation of Pulse Coded Neural Networks , 1994 .

[52]  T. Poggio,et al.  Biophysics of Computation: Neurons, Synapses and Membranes , 1984 .

[53]  Bernabe Linares-Barranco,et al.  A CMOS Implementation of Fitzhugh-Nagumo Neuron Model , 1990, ESSCIRC '90: Sixteenth European Solid-State Circuits Conference.

[54]  A. E. Owen,et al.  Amorphous silicon analogue memory devices , 1989 .

[55]  Simon M. Tam,et al.  Implementation and performance of an analog nonvolatile neural network , 1993 .

[56]  Frances A. Edwards,et al.  LTP — a structural model to explain the inconsistencies , 1995, Trends in Neurosciences.

[57]  T. Teyler,et al.  Long-term potentiation. , 1987, Annual review of neuroscience.

[58]  T. Delbruck.  Silicon retina with correlation-based, velocity-tuned pixels , 1993 .

[59]  R. Douglas,et al.  A silicon neuron , 1991, Nature.

[60]  Dimitris Anastassiou,et al.  Switched-capacitor neural networks , 1987 .

[61]  Y Yarom,et al.  Physiology, morphology and detailed passive models of guinea‐pig cerebellar Purkinje cells. , 1994, The Journal of physiology.

[62]  Bartlett W. Mel,et al.  Information Processing in Dendritic Trees , 1994, Neural Computation.

[63]  C. Stevens,et al.  An evaluation of causes for unreliability of synaptic transmission. , 1994, Proceedings of the National Academy of Sciences of the United States of America.

[64]  Andreas G. Andreou,et al.  Current-mode subthreshold MOS implementation of the Herault-Jutten autoadaptive network , 1992 .

[65]  Mohammed Ismail,et al.  Analog VLSI Implementation of Neural Systems , 2011, The Kluwer International Series in Engineering and Computer Science.

[66]  Tadashi Shibata,et al.  A functional MOS transistor featuring gate-level weighted sum and threshold operations , 1992 .

[67]  R. Nicoll,et al.  An essential role for postsynaptic calmodulin and protein kinase activity in long-term potentiation , 1989, Nature.

[68]  Z. Wang.  Novel electronically-controlled floating resistors using MOS transistors operating in saturation , 1991 .

[69]  Lawrence D. Jackel,et al.  Recent developments of electronic neural nets in North America , 1993, J. VLSI Signal Process..

[70]  Misha Mahowald,et al.  A silicon model of early visual processing , 1993, Neural Networks.

[71]  H Korn,et al.  Transformation of binomial input by the postsynaptic membrane at a central synapse. , 1984, Science.

[72]  David P. M. Northmore,et al.  Switched-capacitor neuromorphs with wide-range variable dynamics , 1995, IEEE Trans. Neural Networks.

[73]  José Luis Huertas,et al.  A CMOS analog adaptive BAM with on-chip learning and weight refreshing , 1993, IEEE Trans. Neural Networks.

[74]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[75]  Paul E. Hasler,et al.  Single transistor learning synapse with long term storage , 1995, Proceedings of ISCAS'95 - International Symposium on Circuits and Systems.

[76]  M M Merzenich,et al.  Temporal information transformed into a spatial code by a neural network with realistic properties , 1995, Science.

[77]  J. L. Martínez,et al.  Long-term potentiation and learning. , 1996, Annual review of psychology.

[78]  Bing J. Sheu,et al.  Analog floating-gate synapses for general-purpose VLSI neural computation , 1991 .

[79]  Idan Segev,et al.  Compartmental models of complex neurons , 1989 .

[80]  Alan F. Murray,et al.  Pulse-stream VLSI neural networks mixing analog and digital techniques , 1991, IEEE Trans. Neural Networks.

[81]  D. Shelton,et al.  Membrane resistivity estimated for the purkinje neuron by means of a passive computer model , 1985, Neuroscience.

[82]  D. Thurbon,et al.  Electrotonic profiles of interneurons in stratum pyramidale of the CA1 region of rat hippocampus. , 1994, Journal of neurophysiology.

[83]  T. Poggio,et al.  A New Approach to Synaptic Interactions , 1978 .

[84]  H. Shinohara,et al.  A refreshable analog VLSI neural network chip with 400 neurons and 40 K synapses , 1992 .

[85]  Misha Anne Mahowald,et al.  VLSI analogs of neuronal visual processing: a synthesis of form and function , 1992 .

[86]  Ralph Etienne-Cummings,et al.  An analog neural computer with modular architecture for real-time dynamic computations , 1992 .

[87]  Idan Segev,et al.  The theoretical foundation of dendritic function: Selected papers of Wilfrid Rall with commentaries , 1994 .

[88]  T. Poggio,et al.  Multiplying with synapses and neurons , 1992 .

[89]  Gordon M. Shepherd,et al.  Canonical neurons and their computational organization , 1992 .

[90]  B. Hille.  Ionic channels of excitable membranes , 2001 .

[91]  T. Murphy,et al.  Visualization of quantal synaptic transmission by dendritic calcium imaging. , 1994, Science.