A graph-theoretic approach to multitasking

A key feature of neural network architectures is their ability to support the simultaneous interaction among large numbers of units in the learning and processing of representations. However, how the richness of such interactions trades off against the ability of a network to simultaneously carry out multiple independent processes -- a salient limitation in many domains of human cognition -- remains largely unexplored. In this paper we use a graph-theoretic analysis of network architecture to address this question, where tasks are represented as edges in a bipartite graph $G=(A \cup B, E)$. We define a new measure of multitasking capacity of such networks, based on the assumptions that tasks that \emph{need} to be multitasked rely on independent resources, i.e., form a matching, and that tasks \emph{can} be multitasked without interference if they form an induced matching. Our main result is an inherent tradeoff between the multitasking capacity and the average degree of the network that holds \emph{regardless of the network architecture}. These results are also extended to networks of depth greater than $2$. On the positive side, we demonstrate that networks that are random-like (e.g., locally sparse) can have desirable multitasking properties. Our results shed light into the parallel-processing limitations of neural systems and provide insights that may be useful for the analysis and design of parallel architectures.

[1]  Jonathan D. Cohen,et al.  Controlled vs. Automatic Processing: A Graph-Theoretic Approach to the Analysis of Serial vs. Parallel Processing in Neural Network Architectures , 2016, CogSci.

[2]  János Komlós,et al.  Convergence results in an associative memory model , 1988, Neural Networks.

[3]  P. Csikvári Lower matching conjecture, and a new proof of Schrijver's and Gurvits's theorems , 2014, 1406.0766.

[4]  D. Kahneman,et al.  Attention and Effort , 1973 .

[5]  D. Falikman Proof of the van der Waerden conjecture regarding the permanent of a doubly stochastic matrix , 1981 .

[6]  James L. McClelland,et al.  On the control of automatic processes: a parallel distributed processing account of the Stroop effect. , 1990, Psychological review.

[7]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[8]  Nir Shavit,et al.  The big data challenges of connectomics , 2014, Nature Neuroscience.

[9]  László Pyber,et al.  Regular subgraphs of dense graphs , 1985, Comb..

[10]  N. Linial,et al.  Expander Graphs and their Applications , 2006 .

[11]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[12]  Vijay V. Vazirani,et al.  Approximation Algorithms , 2001, Springer Berlin Heidelberg.

[13]  Razvan Pascanu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[14]  Christos H. Papadimitriou,et al.  Worst-case Equilibria , 1999, STACS.

[15]  G. Carpenter,et al.  Behavioral and Brain Sciences , 1999 .

[16]  Alexander Schrijver,et al.  Counting 1-Factors in Regular Bipartite Graphs , 1998, J. Comb. Theory B.

[17]  U. Feige,et al.  Generalized Girth Problems in Graphs and Hypergraphs , 2013 .

[18]  Vojtech Rödl,et al.  Dense Graphs without 3-Regular Subgraphs , 1995, J. Comb. Theory, Ser. B.

[19]  Leslie G. Valiant,et al.  Circuits of the mind , 1994 .

[20]  Jonathan D. Cohen,et al.  Multitasking Capability Versus Learning Efficiency in Neural Network Architectures , 2017, CogSci.

[21]  Axthonv G. Oettinger,et al.  IEEE Transactions on Information Theory , 1998 .

[22]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[23]  Noga Alon,et al.  Nearly complete graphs decomposable into large induced matchings and their applications , 2011, STOC '12.

[24]  Jonathan D. Cohen,et al.  The Expected Value of Control: An Integrative Theory of Anterior Cingulate Cortex Function , 2013, Neuron.

[25]  G. Kane Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[26]  R. K. Simpson Nature Neuroscience , 2022 .

[27]  S. Grossberg,et al.  Psychological Review , 2003 .

[28]  H. E. Kuhn,et al.  BULLETIN OF THE AMERICAN MATHEMATICAL SOCIETY, , 2007 .

[29]  Uriel Feige,et al.  Approximating Maximum Edge Coloring in Multigraphs , 2002, APPROX.

[30]  Noga Alon A Note on Degenerate and Spectrally Degenerate Graphs , 2013, J. Graph Theory.

[31]  M. Botvinick,et al.  Conflict monitoring and cognitive control. , 2001, Psychological review.

[32]  Aravaipa Canyon Basin,et al.  Volume 3 , 2012, Journal of Diabetes Investigation.

[33]  J D Cohen,et al.  Multitasking versus multiplexing: Toward a normative account of limitations in the simultaneous execution of control-demanding behaviors , 2014, Cognitive, affective & behavioral neuroscience.

[34]  Walter Schneider,et al.  Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. , 1977 .

[35]  Toniann Pitassi,et al.  Proceedings of the forty-fourth annual ACM symposium on Theory of computing , 2012, STOC 2012.

[36]  Aditya Bhaskara,et al.  Provable Bounds for Learning Some Deep Representations , 2013, ICML.

[37]  H. Pashler Dual-task interference in simple tasks: data and theory. , 1994, Psychological bulletin.

[38]  Tim Roughgarden,et al.  How bad is selfish routing? , 2000, Proceedings 41st Annual Symposium on Foundations of Computer Science.

[39]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[40]  Walter Schneider,et al.  Controlled and Automatic Human Information Processing: 1. Detection, Search, and Attention. , 1977 .

[41]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[42]  Imrich Chlamtac,et al.  On Broadcasting in Radio Networks - Problem Analysis and Protocol Design , 1985, IEEE Transactions on Communications.

[43]  John R. Anderson The Architecture of Cognition , 1983 .

[44]  Angela L. Duckworth,et al.  An opportunity cost model of subjective effort and task performance. , 2013, The Behavioral and brain sciences.

[45]  Nathan Linial,et al.  On the uniform-traffic capacity of single-hop interconnections employing shared directional multichannels , 1993, IEEE Trans. Inf. Theory.

[46]  Prasad Tetali,et al.  Matchings and independent sets of a fixed size in regular graphs , 2009, J. Comb. Theory, Ser. A.

[47]  V. Sós,et al.  On a problem of K. Zarankiewicz , 1954 .

[48]  Noga Alon,et al.  List Coloring of Random and Pseudo-Random Graphs , 1999, Comb..

[49]  R. Shiffrin,et al.  Controlled and automatic human information processing: I , 1977 .

[50]  G. Egorychev The solution of van der Waerden's problem for permanents , 1981 .

[51]  Benjamin Naumann The Architecture Of Cognition , 2016 .

[52]  October I Physical Review Letters , 2022 .