Natural constraints explain working memory capacity limitations in sensory-cognitive models

The limited capacity of the brain to retain information in working memory has been well-known and studied for decades, yet the root of this limitation remains unclear. Here we built sensory-cognitive neural network models of working memory that perform tasks using raw visual stimuli. Contrary to intuitions that working memory capacity limitation stems from memory or cognitive constraints, we found that pre-training the sensory region of our models with natural images imposes sufficient constraints on models to exhibit a wide range of human-like behaviors in visual working memory tasks designed to probe capacity. Examining the neural mechanisms in our model reveals that capacity limitation mainly arises in a bottom-up manner. Our models offer a principled and functionally grounded explanation for the working memory capacity limitation without parameter fitting to behavioral data or much hyperparameter tuning. This work highlights the importance of developing models with realistic sensory processing even when investigating memory and other high-level cognitive phenomena.

[1]  Jaime de la Rocha,et al.  Recurrent networks endowed with structural priors explain suboptimal animal behavior , 2023, Current Biology.

[2]  T. Mrsic-Flogel,et al.  Cortical feedback loops bind distributed representations of working memory , 2022, Nature.

[3]  Brad Wyble,et al.  A model of working memory for latent representations , 2021, Nature Human Behaviour.

[4]  Thomas Parr,et al.  The Predictive Brain Must Have a Limitation in Short-Term Memory Capacity , 2021, Current Directions in Psychological Science.

[5]  Daniel L. K. Yamins,et al.  Shallow Unsupervised Models Best Predict Neural Responses in Mouse Visual Cortex , 2021 .

[6]  Daniel L. K. Yamins,et al.  Goal-Driven Recurrent Neural Network Models of the Ventral Visual Stream , 2021 .

[7]  Michael C. Frank,et al.  Unsupervised neural network models of the ventral visual stream , 2020, Proceedings of the National Academy of Sciences.

[8]  Robert Taylor,et al.  Stochastic sampling provides a unifying account of visual working memory limits , 2020, Proceedings of the National Academy of Sciences.

[9]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[10]  Timothy F. Brady,et al.  Psychophysical scaling reveals a unified theory of visual memory strength , 2018, Nature Human Behaviour.

[11]  Madhura R. Joglekar,et al.  Task representations in neural networks trained to perform many cognitive tasks , 2019, Nature Neuroscience.

[12]  Flora Bouchacourt,et al.  A Flexible Model of Working Memory , 2018, Neuron.

[13]  Earl K. Miller,et al.  Working Memory 2.0 , 2018, Neuron.

[14]  Jonas Kubilius,et al.  Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like? , 2018, bioRxiv.

[15]  Wei Ji Ma,et al.  A resource-rational theory of set size effects in human visual working memory , 2018, eLife.

[16]  In-So Kweon,et al.  CBAM: Convolutional Block Attention Module , 2018, ECCV.

[17]  P. Roelfsema,et al.  The Distributed Nature of Working Memory , 2017, Trends in Cognitive Sciences.

[18]  Mikhail Katkov,et al.  Synaptic Correlates of Working Memory Capacity , 2017, Neuron.

[19]  Timothy F. Brady,et al.  Working memory is not fixed-capacity: More active storage capacity for real-world objects than for simple stimuli , 2016, Proceedings of the National Academy of Sciences.

[20]  Christopher D. Harvey,et al.  Recurrent Network Models of Sequence Generation and Memory , 2016, Neuron.

[21]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[22]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Paul M Bays,et al.  Noise in Neural Populations Accounts for Errors in Working Memory , 2014, The Journal of Neuroscience.

[24]  W. Ma,et al.  Changing concepts of working memory , 2014, Nature Neuroscience.

[25]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[26]  E. Vogel,et al.  Visual working memory capacity: from psychophysics and neurobiology to individual differences , 2013, Trends in Cognitive Sciences.

[27]  Wei Ji Ma,et al.  No Evidence for an Item Limit in Change Detection , 2013, PLoS Comput. Biol..

[28]  Frank Tong,et al.  Introspective judgments predict the precision and likelihood of successful maintenance of visual working memory. , 2012, Journal of vision.

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Wei Ji Ma,et al.  Variability in encoding precision accounts for visual short-term memory limitations , 2012, Proceedings of the National Academy of Sciences.

[31]  Markus Siegel,et al.  Neural substrates of cognitive capacity limitations , 2011, Proceedings of the National Academy of Sciences.

[32]  Paul M Bays,et al.  Dynamic Updating of Working Memory Resources for Visual Objects , 2011, The Journal of Neuroscience.

[33]  Markus Siegel,et al.  Phase-dependent neuronal coding of objects in short-term memory , 2009, Proceedings of the National Academy of Sciences.

[34]  Paul M Bays,et al.  The precision of visual working memory is set by allocation of a shared resource. , 2009, Journal of vision.

[35]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Jesper Tegnér,et al.  Mechanism for top-down control of working memory capacity , 2009, Proceedings of the National Academy of Sciences.

[37]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[38]  Paul M Bays,et al.  Dynamic Shifts of Limited Working Memory Resources in Human Vision , 2008, Science.

[39]  S. Luck,et al.  Discrete fixed-resolution representations in visual working memory , 2008, Nature.

[40]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[41]  E. Vogel,et al.  PSYCHOLOGICAL SCIENCE Research Article Visual Working Memory Represents a Fixed Number of Items Regardless of Complexity , 2022 .

[42]  W. Ma,et al.  A detection theory account of change detection. , 2004, Journal of vision.

[43]  Maro G. Machizawa,et al.  Neural activity predicts individual differences in visual working memory capacity , 2004, Nature.

[44]  J. Jay Todd,et al.  Capacity limit of visual short-term memory in human posterior parietal cortex , 2004, Nature.

[45]  B. Baars,et al.  How conscious experience and working memory interact , 2003, Trends in Cognitive Sciences.

[46]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[47]  Edward K. Vogel,et al.  The capacity of visual working memory for features and conjunctions , 1997, Nature.

[48]  J. Elman.  Learning and development in neural networks: the importance of starting small , 1993, Cognition.

[49]  Peter M. Duppenthaler.  Maturational Constraints on Language Learning , 1990 .

[50]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .