The Nature of Mental Images - An Integrative Computational Theory

The Nature of Mental Images – An Integrative Computational Theory Jan Frederik Sima (sima@sfbtr8.uni-bremen.de) SFB/TR 8 Spatial Cognition, University of Bremen, Germany Quasi-Pictorial Theory Abstract We shed new light on the long-debated question about the na- ture of mental images, that is, the underlying structures and processes, with a new theory of mental imagery. This theory is formalized as a computational cognitive model and provides an integrated account of the three prevalent theories of mental imagery, i.e., the descriptive, the quasi-pictorial, and the enac- tive theory. It does so by offering a consistent explanation for a set of empirical results, which are not plausibly provided by any of the theories individually. We give a brief review of the three theories and summarize their core commitments from a computational modeling perspective. We present a set of em- pirical results, the different explanations offered by the three theories, and deficiencies of their explanations. The proposed theory and model are introduced and the model’s explanatory power is evaluated using the previously identified set of phe- nomena. Keywords: Mental Imagery; Cognitive Modeling; Imagery Debate; Mental Representations Introduction It is a widely accepted assumption, that we cannot prove the nature of the mental structures underlying mental imagery based on behavioral data (Anderson, 1978). Nevertheless, the large and growing stock of empirical data forces refine- ment or extension of existing theories to plausibly explain as many results as possible. Recent findings include results from eye tracking studies and brain imaging methods, for ex- ample. The three major accounts of explaining mental im- agery, namely the descriptive theory (e.g., Pylyshyn, 2002), the quasi-pictorial theory (e.g., Kosslyn, 1994), and the en- active theory (e.g., Thomas, 1999), gain their theoretical rel- evance by explaining certain experimental results which the other theories cannot with the same degree of plausibility. In this paper, we will focus on a set of such distinguishing ex- periments. We will show how our new theory, that is im- plemented as a computational cognitive model, explains, in particular, these empirical results and offers an integrated ac- count of the three different approaches. We will first identify the core commitments of the existing theories and the relevant experimental studies. We will then describe our theory, its main assumptions, and present the re- sulting computational model. Afterwards, we will be able to evaluate the new theory against the previously selected exper- imental results. Theories of Mental Imagery We focus on the identification of the core commitments of each theory and do not aim at offering a comprehensive overview, as each of the following theories has several pro- ponents, who themselves shape and interpret the respective theory in different ways. Particularly, we emphasize the im- plications of these commitments for a potential computational implementation. An example from Kosslyn (1980) illustrates the basic idea of this theory as follows: the answer whether a fox has pointy or round-shaped ears is solved by retrieving the necessary (en- coded) visual information from long-term memory and gen- erating a “picture-like” representation of a fox in an internal representation structure, called the visual buffer. This mental image is then inspected to make the information conscious, i.e., the answer is read off the depictive representation. This means, a spatio-analogical mental representation is holding depictive visual information during imagery. We identify three core commitments of the quasi-pictorial theory: 1) the existence and usage of a visual buffer (i.e., a spatio-analogical representation structure), 2) the generation of a percept-like activation in this buffer, and 3) the active in- spection of this “percept” to extract information by processes partly shared with visual perception. Many different issues have been raised regarding the idea of quasi-pictures being mentally inspected by processes shared with visual percep- tion (e.g., Slezak, 1995; Pylyshyn, 2002; Thomas, 1999). A general problem of the theory, that becomes unavoidable in a computational implementation, is the lack of formalization of the apparent ambiguous nature of mental images: Empirical data indicates that mental images are much like actual im- ages in some respects (e.g., linear scanning time, see Denis & Kosslyn, 1999), but different in other respects (e.g., difficulty of reinterpretation, see Slezak, 1995). Descriptive Theory The descriptive or propositional theory is most prominently defended by Pylyshyn (e.g., Pylyshyn, 2002) as the null hy- pothesis contrasting the quasi-pictorial theory. The main point of the theory is the rejection of a spatio-analogical, i.e., “depictive”, representation and the claim that the format of the representations underlying mental imagery are purely propositional. Thus, proponents of this theory interpret em- pirical data that potentially contradicts a picture-like repre- sentation as arguments for the descriptive theory. The de- scriptive theory was extended with the concept of tacit knowl- edge (Pylyshyn, 1981) to explain (at that time) new chrono- metric data, e.g., in mental rotation or mental scanning tasks, which arguably pose strong support to the idea of an analog- ical “percept”. It is hypothesized that humans use their tacit knowledge of what it would be like to see something in actual visual perception during certain mental imagery tasks to pro- duce the linear reaction time patterns during mental scanning, for example. We conclude the core commitment of the descriptive the- ory to be the involvement of only non-analogical, proposi- tional representation structures in mental imagery. From a