UC Merced Proceedings of the Annual Meeting of the Cognitive Science Society Title An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity – Precision Tradeoffs Permalink

An Ideal Observer Model of Visual Short-Term Memory Predicts Human Capacity–Precision Tradeoffs Chris R. Sims Robert A. Jacobs David C. Knill (csims@cvs.rochester.edu) (robbie@bcs.rochester.edu) (knill@cvs.rochester.edu) Department of Brain and Cognitive Sciences & Center for Visual Science University of Rochester, Rochester, NY Abstract p(x) p(xs | x) We develop an ideal observer model of human visual short- term memory. Compared to previous models that have posited constraints on memory performance intended solely to ac- count for observed phenomena, in the present research we de- rive the expected behavior of an optimally performing, but limited-capacity memory system. We develop our model us- ing rate–distortion theory, a branch of information theory that provides optimal bounds on the accuracy of information trans- mission subject to a fixed capacity. The resulting model pro- vides a task-independent and theoretically motivated definition of visual memory capacity and yields novel predictions regard- ing human performance. These predictions are quantitatively evaluated in an empirical study. We also demonstrate that our ideal observer model encompasses two existing, but competing accounts of VSTM as special cases. Keywords: Ideal observer analysis, VSTM, information the- ory, rate–distortion theory Information source Sensory signal xs VSTM Memory encoding / storage p(x ˆ | xm ) Recovered signal xm Optimal encoder & decoder Memory decoding / retrieval Figure 1: A schematic diagram of the ideal observer model of VSTM. Introduction but rather only serves to account for observed empirical phe- nomena. Second, the definition of capacity appears largely task-dependent, and it is therefore difficult to form predictions for human performance in different tasks or conditions. In the case of the discrete slot model, there is no strong theoretical justification for determining what visual features or objects can or cannot occupy a single slot, and in the continuous re- source model, the nature of the resource that is being divided is only abstractly specified. Visual short-term memory (VSTM)—defined as the ability to store task-relevant visual information in a rapidly acces- sible and easily manipulated form—is a central component of nearly all human activities. Given its importance, it is per- haps surprising that the capacity of this system is severely lim- ited. Numerous investigations of VSTM performance have revealed that we can only accurately store a surprisingly small number of visual objects or features (for a review, see Luck, In recent years there have been numerous attempts to de- fine and quantify what is meant by VSTM capacity. Until recently, the predominant view has held that capacity is lim- ited to a small, fixed number of visual objects (typically as- sumed to be 4) stored in discrete “slots” (Awh, Barton, & Vo- gel, 2007; Luck & Vogel, 1997; Vogel, Woodman, & Luck, 2001; Cowan & Rouder, 2009). Taking a different approach, Bays and colleagues (Bays, Catalao, & Husain, 2009; Bays & Husain, 2008) explored how the precision of features en- coded in visual working memory may change as a function of the number of features that are concurrently stored. Based on the finding that memory precision appears to decrease even when as few as 2 items are encoded, the authors proposed that VSTM capacity consists of a single, continuous resource that must be divided among items stored in working memory. While both the discrete slot and continuous resource mod- els are able to account for a number of empirical findings, both are ultimately unsatisfactory as complete theories of hu- man VSTM. First, in the case of both models the nature of the capacity limit is somewhat arbitrary: the hypothesized capac- ity limit does not emerge from a principled theoretical basis, The Ideal Observer Model of VSTM In this section we derive an ideal observer model of visual short-term memory. We show that from an information- theoretic perspective, our ideal observer model is optimally efficient in that it minimizes a particular measure of memory distortion subject to a fixed capacity limit. The resulting model makes several important contributions to the literature on VSTM. First, whereas previous models have postulated abstract or task-dependent definitions of vi- sual memory capacity, the ideal observer model provides a quantitative definition of capacity that is task-independent and easily interpreted. This enables results obtained in one ex- periment to generate predictions for performance in another. Second, since our model exhibits provably optimal perfor- mance subject to a fixed capacity, it can be used to obtain an assumption-free estimate of the minimum capacity of hu- man VSTM. Finally, we demonstrate that our ideal observer model subsumes two existing models of VSTM: a recent ver- sion of the discrete slot model (Cowan & Rouder, 2009), and the continuous resource model (Bays & Husain, 2008).