Multitasking in RNN: an analysis exploring the combination of simple tasks

The brain and artificial neural networks are capable of performing multiple tasks. The mechanisms through which simultaneous tasks are performed by the same set of units in the brain are not yet entirely clear. Such systems can be modular or mixed selective through some variables such as sensory stimulus. Recurrent neural networks can help to a better understanding of those mechanisms. Based on simple tasks studied previously in Jarne 2020 arXiv Preprint 2005.13074, multitasking networks were trained and analyzed. In present work, a simple model that can perform multiple tasks using a contextual signal was studied, trying to illuminate mechanisms similar to those that could occur in biological brains. Backpropagation through time allows training networks with multitasking, but the realizations obtained are not unique. Different realizations for the same set of tasks are possible. Here the analysis of the dynamics and emergent behavior of their units is presented. The goal is to try to describe better the models used to describe different processes in the cortex.

[1]  Mounir Boukadoum,et al.  Mechanisms Gating the Flow of Information in the Cortex: What They Might Look Like and What Their Uses may be , 2010, Front. Comput. Neurosci..

[2]  Clay M. Thompson,et al.  Image processing toolbox [for use with Matlab] , 1995 .

[3]  R. Romo,et al.  Neuronal Population Coding of Parametric Working Memory , 2010, The Journal of Neuroscience.

[4]  Timoteo Carletti,et al.  Structure and dynamical behaviour of non-normal networks. , 2018, 1803.11542.

[5]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[6]  Michael W. Cole,et al.  How to study the neural mechanisms of multiple tasks , 2019, Current Opinion in Behavioral Sciences.

[7]  Omri Barak,et al.  Recurrent neural networks as versatile tools of neuroscience research , 2017, Current Opinion in Neurobiology.

[8]  Matthew T. Kaufman,et al.  A neural network that finds a naturalistic solution for the production of muscle activity , 2015, Nature Neuroscience.

[9]  Krishna V. Shenoy,et al.  Computation Through Neural Population Dynamics. , 2020, Annual review of neuroscience.

[10]  Byron M. Yu,et al.  Dimensionality reduction for large-scale neural recordings , 2014, Nature Neuroscience.

[11]  Daniel Durstewitz,et al.  Attracting Dynamics of Frontal Cortex Ensembles during Memory-Guided Decision-Making , 2011, PLoS Comput. Biol..

[12]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[13]  Matthew T. Kaufman,et al.  A category-free neural population supports evolving demands during decision-making , 2014, Nature Neuroscience.

[14]  Haim Sompolinsky,et al.  Coherent chaos in a recurrent neural network with structured connectivity , 2018, PLoS Comput. Biol..

[15]  Devika Narain,et al.  Flexible timing by temporal scaling of cortical responses , 2017, Nature Neuroscience.

[16]  David Sussillo,et al.  Neural circuits as computational dynamical systems , 2014, Current Opinion in Neurobiology.

[17]  Cecilia Jarne,et al.  The dynamics of Recurrent Neural Networks trained for temporal tasks and the eigenvalue spectrum , 2020 .

[18]  Guillaume Hennequin,et al.  Neuroscience out of control: control-theoretic perspectives on neural circuit dynamics , 2019, Current Opinion in Neurobiology.

[19]  David Sussillo,et al.  Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks , 2013, Neural Computation.

[20]  Abigail A. Russo,et al.  Motor Cortex Embeds Muscle-like Commands in an Untangled Population Response , 2018, Neuron.

[21]  Xiao-Jing Wang,et al.  Task representations in neural networks trained to perform many cognitive tasks , 2019, Nature Neuroscience.

[22]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[23]  N. Caticha,et al.  On-line learning in the committee machine , 1995 .