Dynamics of random recurrent networks with correlated low-rank structure

A given neural network in the brain is involved in many different tasks. This implies that, when considering a specific task, the network's connectivity contains a component which is related to the task and another component which can be considered random. Understanding the interplay between the structured and random components, and their effect on network dynamics and functionality is an important open question. Recent studies addressed the co-existence of random and structured connectivity, but considered the two parts to be uncorrelated. This constraint limits the dynamics and leaves the random connectivity non-functional. Algorithms that train networks to perform specific tasks typically generate correlations between structure and random connectivity. Here we study nonlinear networks with correlated structured and random components, assuming the structure to have a low rank. We develop an analytic framework to establish the precise effect of the correlations on the eigenvalue spectrum of the joint connectivity. We find that the spectrum consists of a bulk and multiple outliers, whose location is predicted by our theory. Using mean-field theory, we show that these outliers directly determine both the fixed points of the system and their stability. Taken together, our analysis elucidates how correlations allow structured and random connectivity to synergistically extend the range of computations available to networks.

[1]  J. Ginibre.  Statistical Ensembles of Complex, Quaternion, and Real Matrices , 1965 .

[2]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[3]  Sompolinsky,et al.  Spin-glass models of neural networks. , 1985, Physical review. A, General physics.

[4]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[5]  Misha Tsodyks,et al.  Chaos in Highly Diluted Neural Networks , 1991 .

[6]  P. Mason,et al.  One step back , 1993, Nature.

[7]  H. Sompolinsky,et al.  Theory of orientation tuning in visual cortex. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[8]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[9]  D. Harville.  Matrix Algebra From a Statistician's Perspective , 1998 .

[10]  Xiao-Jing Wang,et al.  Probabilistic Decision Making by Slow Reverberation in Cortical Circuits , 2002, Neuron.

[11]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[12]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[13]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[14]  L. Abbott,et al.  Eigenvalue spectra of random matrices for neural networks. , 2006, Physical review letters.

[15]  Peter E. Latham,et al.  A Balanced Memory Network , 2007, PLoS Comput. Biol..

[16]  Xiao-Jing Wang,et al.  Mean-Driven and Fluctuation-Driven Persistent Activity in Recurrent Networks , 2007, Neural Computation.

[17]  L. F. Abbott,et al.  Generating Coherent Patterns of Activity from Chaotic Neural Networks , 2009, Neuron.

[18]  L. Abbott,et al.  Stimulus-dependent suppression of chaos in recurrent neural networks. , 2009, Physical review. E, Statistical, nonlinear, and soft matter physics.

[19]  T. Tao.  Outliers in the spectrum of iid matrices with bounded rank perturbations , 2010 .

[20]  F L Metz,et al.  Spectra of sparse non-hermitian random matrices: an analytical solution. , 2012, Physical review letters.

[21]  G. Wainrib,et al.  Topological and dynamical complexity of random neural networks. , 2012, Physical review letters.

[22]  L. Abbott,et al.  From fixed points to chaos: Three models of delayed discrimination , 2013, Progress in Neurobiology.

[23]  Xiao-Jing Wang,et al.  The importance of mixed selectivity in complex cognitive tasks , 2013, Nature.

[24]  Kenneth D Miller,et al.  Properties of networks with partially structured and partially random connectivity. , 2013, Physical review. E, Statistical, nonlinear, and soft matter physics.

[25]  Merav Stern,et al.  Transition to chaos in random networks with cell-type-specific connectivity. , 2014, Physical review letters.

[26]  Izaak Neri,et al.  Eigenvalue Outliers of Non-Hermitian Random Matrices with a Local Tree Structure. , 2016, Physical review letters.

[27]  Peter Ford Dominey,et al.  Reservoir Computing Properties of Neural Dynamics in Prefrontal Cortex , 2016, PLoS Comput. Biol..

[28]  Alexander Rivkind,et al.  Local Dynamics in Trained Recurrent Neural Networks. , 2015, Physical review letters.

[29]  Brent Doiron,et al.  Circuit-based models of shared variability in cortical networks , 2017, bioRxiv.

[30]  Francesca Mastrogiuseppe,et al.  Linking Connectivity, Dynamics, and Computations in Low-Rank Recurrent Neural Networks , 2017, Neuron.

[31]  Kunihiko Kaneko,et al.  Formation of Dominant Mode by Evolution in Biological Systems , 2017, bioRxiv.

[32]  Sean Escola,et al.  A model of flexible motor sequencing through thalamic control of cortical dynamics , 2019, bioRxiv.

[33]  Omri Barak,et al.  One Step Back, Two Steps Forward: Interference and Learning in Recurrent Neural Networks , 2018, Neural Computation.

[34]  T. Rogers,et al.  Spectral theory of sparse non-Hermitian random matrices , 2018, Journal of Physics A: Mathematical and Theoretical.

[35]  Xiao-Jing Wang,et al.  Task representations in neural networks trained to perform many cognitive tasks , 2019, Nature Neuroscience.

[36]  Francesca Mastrogiuseppe,et al.  A Geometrical Analysis of Global Stability in Trained Feedback Networks , 2019, Neural Computation.

[37]  Alexander Rivkind,et al.  Scale free topology as an effective feedback system , 2019, bioRxiv.