Model of birdsong learning based on gradient estimation by dynamic perturbation of neural conductances.

We propose a model of songbird learning that focuses on avian brain areas HVC and RA, involved in song production, and area LMAN, important for generating song variability. Plasticity at HVC --> RA synapses is driven by hypothetical "rules" depending on three signals: activation of HVC --> RA synapses, activation of LMAN --> RA synapses, and reinforcement from an internal critic that compares the bird's own song with a memorized template of an adult tutor's song. Fluctuating glutamatergic input to RA from LMAN generates behavioral variability for trial-and-error learning. The plasticity rules perform gradient-based reinforcement learning in a spiking neural network model of song production. Although the reinforcement signal is delayed, temporally imprecise, and binarized, the model learns in a reasonable amount of time in numerical simulations. Varying the number of neurons in HVC and RA has little effect on learning time. The model makes specific predictions for the induction of bidirectional long-term plasticity at HVC --> RA synapses.

[1]  P. Marler,et al.  Culturally Transmitted Patterns of Vocal Behavior in Sparrows , 1964, Science.

[2]  M. Konishi The role of auditory feedback in the control of vocalization in the white-crowned sparrow. , 1965, Zeitschrift fur Tierpsychologie.

[3]  K. Immelmann Song development in the zebra finch and other estrildid finches , 1969 .

[4]  Robert W. Warner,et al.  The anatomy of the syrinx in passerine birds , 1972 .

[5]  F. Nottebohm,et al.  Central control of song in the canary, Serinus canarius , 1976, The Journal of comparative neurology.

[6]  Philip H. Price Developmental determinants of structure in zebra finch song. , 1979 .

[7]  M.G. Bellanger,et al.  Digital processing of speech signals , 1980, Proceedings of the IEEE.

[8]  M. Gurney,et al.  Hormonal control of cell form and number in the zebra finch song system , 1981, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[9]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[10]  A. Arnold,et al.  Forebrain lesions disrupt development but not maintenance of song in passerine birds. , 1984, Science.

[11]  P. Anandan,et al.  Pattern-recognizing stochastic learning automata , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[12]  S. Nowicki,et al.  Vocal tract resonances in oscine bird sound production: evidence from birdsongs in a helium atmosphere , 1987, Nature.

[13]  Neville H Fletcher,et al.  Bird song—A quantitative acoustic model , 1988 .

[14]  F. Nottebohm,et al.  Effect of testosterone on input received by an identified neuron type of the canary song system: a Golgi/electron microscopy/degeneration study , 1988, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[15]  I. Titze The physics of small-amplitude oscillation of the vocal folds. , 1988, The Journal of the Acoustical Society of America.

[16]  D. Vicario,et al.  Brain pathways for learned and unlearned vocalizations differ in zebra finches , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[17]  Thomas Kailath,et al.  Model-free distributed learning , 1990, IEEE Trans. Neural Networks.

[18]  F. Nottebohm,et al.  A comparative study of the behavioral deficits following lesions of various parts of the zebra finch song system: implications for vocal learning , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[19]  Evon M. O. Abu-Taieh,et al.  Comparative Study , 2020, Definitions.

[20]  A. Arnold,et al.  The development of afferent projections to the robust archistriatal nucleus in male zebra finches: a quantitative electron microscopic study , 1991, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[21]  R. Mooney,et al.  Synaptic basis for developmental plasticity in a birdsong nucleus , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[22]  Gert Cauwenberghs,et al.  A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization , 1992, NIPS.

[23]  A. Doupe A neural circuit specialized for vocal learning , 1993, Current Opinion in Neurobiology.

[24]  J. Wild Descending projections of the songbird nucleus robustus archistriatalis , 1993, The Journal of comparative neurology.

[25]  K. Chergui,et al.  Uptake of Dopamine Released by Impulse Flow in the Rat Mesolimbic and Striatal Systems In Vivo , 1995, Journal of neurochemistry.

[26]  A. C. Yu,et al.  Temporal Hierarchical Control of Singing in Birds , 1996, Science.

[27]  H Sakaguchi,et al.  Developmental changes in axon terminals visualized by immunofluorescence for the growth-associated protein, GAP-43, in the robust nucleus of the archistriatum of the zebra finch. , 1996, Brain research. Developmental brain research.

[28]  F. Goller,et al.  A new mechanism of sound generation in songbirds. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[29]  J. Wild,et al.  Neural pathways for the control of birdsong production. , 1997, Journal of neurobiology.

[30]  H. Scheich,et al.  Dopaminergic and Serotonergic Neurotransmission Systems Are Differentially Involved in Auditory Cortex Learning: A Long‐Term Microdialysis Study of Metabolites , 1997, Journal of neurochemistry.

[31]  J. Brugge,et al.  Central Auditory Processing and Neural Modeling , 2012, Springer US.

[32]  Bijan Pesaran,et al.  The role of nonlinear dynamics of the syrinx in the vocalizations of a songbird , 1998, Nature.

[33]  T. Sejnowski,et al.  A Computational Model of Birdsong Learning by Auditory Experience and Auditory Feedback , 1998 .

[34]  S. Ribeiro,et al.  Noradrenergic system of the zebra finch brain: Immunocytochemical study of dopamine‐β‐hydroxylase , 1998, The Journal of comparative neurology.

[35]  A. Doupe,et al.  Social context modulates singing-related neural activity in the songbird forebrain , 1999, Nature Neuroscience.

[36]  R. Mooney,et al.  Lesions of an Avian Forebrain Nucleus That Disrupt Song Development Alter Synaptic Connectivity and Transmission in the Vocal Premotor Pathway , 1999, The Journal of Neuroscience.

[37]  F Goller,et al.  The neuromuscular control of birdsong. , 1999, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[38]  U. Alon,et al.  Robustness in bacterial chemotaxis , 1999, Nature.

[39]  A. Doupe,et al.  Singing-Related Neural Activity in a Dorsal Forebrain–Basal Ganglia Circuit of Adult Zebra Finches , 1999, The Journal of Neuroscience.

[40]  D. Perkel,et al.  Two-Stage, Input-Specific Synaptic Maturation in a Nucleus Essential for Vocal Production in the Zebra Finch , 1999, The Journal of Neuroscience.

[41]  O Tchernichovski,et al.  Vocal imitation in zebra finches is inversely related to model abundance. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[42]  A. Doupe,et al.  Interruption of a basal ganglia–forebrain circuit prevents plasticity of learned vocalizations , 2000, Nature.

[43]  T W Troyer,et al.  An associational model of birdsong sensorimotor learning I. Efference copy and the learning of song syllables. , 2000, Journal of neurophysiology.

[44]  M. Gazzaniga,et al.  The new cognitive neurosciences , 2000 .

[45]  T. Sejnowski,et al.  3 3 A Computational Model of Avian Song Learning , 2000 .

[46]  Michael S. Brainard,et al.  Auditory feedback in learning and maintenance of vocal behaviour , 2000, Nature Reviews Neuroscience.

[47]  Peter L. Bartlett,et al.  Infinite-Horizon Policy-Gradient Estimation , 2001, J. Artif. Intell. Res..

[48]  Todd W Troyer,et al.  Birdsong: models and mechanisms , 2001, Current Opinion in Neurobiology.

[49]  Sacha B. Nelson,et al.  Postsynaptic Depolarization Scales Quantal Amplitude in Cortical Pyramidal Neurons , 2001, The Journal of Neuroscience.

[50]  F. Nottebohm,et al.  Dynamics of the Vocal Imitation Process: How a Zebra Finch Learns Its Song , 2001, Science.

[51]  Michael I. Jordan,et al.  Optimal feedback control as a theory of motor coordination , 2002, Nature Neuroscience.

[52]  D. Margoliash Evaluating theories of bird song learning: implications for future directions , 2002, Journal of Comparative Physiology A.

[53]  Michael S. Brainard,et al.  What songbirds teach us about learning , 2002, Nature.

[54]  K. D. Punta,et al.  An ultra-sparse code underlies the generation of neural sequences in a songbird , 2002 .

[55]  F. Johnson,et al.  Quantifying song bout production during zebra finch sensory-motor learning suggests a sensitive period for vocal practice , 2002, Behavioural Brain Research.

[56]  D. Margoliash,et al.  Motor control of birdsong , 2002, Current Opinion in Neurobiology.

[57]  Carel ten Cate,et al.  Pure-tone birdsong by resonance filtering of harmonic overtones , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[58]  H. Seung,et al.  Learning in Spiking Neural Networks by Reinforcement of Stochastic Synaptic Transmission , 2003, Neuron.

[59]  Johan L. van Leeuwen,et al.  Quantitative modelling of the biomechanics of the avian syrinx , 2003 .

[60]  Stephen D. Shea,et al.  Basal Forebrain Cholinergic Modulation of Auditory Activity in the Zebra Finch Song System , 2003, Neuron.

[61]  Xiaohui Xie,et al.  Learning in neural networks by reinforcement of irregular spiking. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[62]  David J Perkel,et al.  Origin of the Anterior Forebrain Pathway , 2004, Annals of the New York Academy of Sciences.

[63]  Ronald J. Williams,et al.  Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning , 2004, Machine Learning.

[64]  Ila R Fiete,et al.  Temporal sparseness of the premotor drive is important for rapid learning in a neural network model of birdsong. , 2004, Journal of neurophysiology.

[65]  David J Perkel,et al.  Songbirds and the Revised Avian Brain Nomenclature , 2004, Annals of the New York Academy of Sciences.

[66]  J Martin Wild,et al.  Functional Neuroanatomy of the Sensorimotor Control of Singing , 2004, Annals of the New York Academy of Sciences.

[67]  Anthony Leonardo,et al.  Experimental test of the birdsong error-correction model. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[68]  Richard Hans Robert Hahnloser,et al.  Neural Mechanisms of Vocal Sequence Generation in the Songbird , 2004, Annals of the New York Academy of Sciences.

[69]  M. Farries The Avian Song System in Comparative Perspective , 2004, Annals of the New York Academy of Sciences.

[70]  A. Doupe,et al.  Contributions of an avian basal ganglia–forebrain circuit to real-time modulation of song , 2005, Nature.

[71]  A. Leonardo,et al.  Ensemble Coding of Vocal Control in Birdsong , 2005, The Journal of Neuroscience.

[72]  Masakazu Konishi,et al.  Neural auditory selectivity develops in parallel with song. , 2005, Journal of neurobiology.

[73]  Aaron S. Andalman,et al.  Vocal Experimentation in the Juvenile Songbird Requires a Basal Ganglia Circuit , 2005, PLoS biology.

[74]  Xiaohui Xie,et al.  Learning Curves for Stochastic Gradient Descent in Linear Feedforward Networks , 2003, Neural Computation.

[75]  Ila R Fiete,et al.  Gradient learning in spiking neural networks by dynamic perturbation of conductances. , 2006, Physical review letters.

[76]  David J Perkel,et al.  Noradrenergic modulation of activity in a vocal control nucleus in vitro. , 2006, Journal of neurophysiology.