An Information Search Model Integrating Visual, Semantic and Memory Processes

An Information Search Model Integrating Visual, Semantic and Memory Processes Myriam Chanceaux (myriam.chanceaux@imag.fr) University of Grenoble, Laboratoire TIMC-IMAG, Domaine de la Merci 38700 La Tronche FRANCE Anne Gu´erin-Dugu´e (anne.guerin@gipsa-lab.inpg.fr ) University of Grenoble, Laboratoire Gipsa-Lab, Domaine universitaire 38402 Saint Martin d’H`eres FRANCE Benoˆit Lemaire (benoit.lemaire@imag.fr) University of Grenoble, Laboratoire TIMC-IMAG, Domaine de la Merci 38700 La Tronche FRANCE Thierry Baccino (baccino@lutin-userlab.fr) Laboratoire LUTIN, Cit´e des sciences et de l’industrie de la Villette 75930 Paris cedex 19 FRANCE Abstract collected during two experiments where participants had to search for information. This study aims at presenting a computational model of visual search including a visual, a semantic and a memory process in order to simulate human behavior during information seeking. We implemented the memory process, which is the most im- portant part of the model based on the Variable Memory Model (Arani, Karwan, & G., 1984; Horowitz, 2006). To compare model and humans, we designed two experiments where par- ticipants were asked to find the word among forty distributed on the display, which best answers a question. The results showed good fits on different features extracted between em- pirical and simulated scanpaths. Keywords: Visual Memory; Information Seeking; Computa- tional Model; Eye Movements; Semantic Similarities Model architecture Introduction Nowadays information seeking, on a Web page for exam- ple, is a very common task. That is why for several years research has been conducted to try to answer this question: what guide user’s attention in this particular task, especially on the web? The literature contains theoretical models of in- formation seeking activity (Marchionini, 1997), especially in electronic documents, and also computational models which simulate navigation between pages, with cognitive architec- tures like ACT-R (Pirolli & Fu, 2003). There are also a lot of studies based on the Feature Integration Theory (Treisman & Gelade, 1980) with models which take into account the vi- sual features of stimuli: colors, orientation, contrast (Itti & Koch, 2000) to determine the most salient part of the stim- ulus. However even if some models take into account the semantic information of the material, in addition to visual in- formation (Navalpakkam & Itti, 2005), experiments and mod- elling are missing in this field. The purpose of this paper is to describe a cognitively plausible model that takes into ac- count semantic and visual features of stimuli when searching for information. This model is an improvement of a simpler one (Chanceaux, Gu´erin-Dugu´e, Lemaire, & Baccino, 2008). It has been implemented and compared to experimental data In this section we will describe our model. This model sim- ulates human eye movements during a simple task of infor- mation seeking involving text and visual features. We devel- oped an architecture in 3 parts, (see Figure 1), corresponding to 3 main cognitive processes involved in a task of informa- tion search. These three processes are respectively related to bottom-up visual information, top-down semantic informa- tion and memory. These processes will be detailed thereafter. The principle of the model is to predict from a fixed location and an history of previous fixations where will be the next fixation. It is assumed that from a given point each location of the image has a visual weight (calculated by the visual pro- cess) and a semantic weight (calculated by the semantic pro- cess). These values are modulated by the process of memory. The basic process of memory is that if a location has already been visited it has less interest than if it never was. At each iteration the location of the fixation is thus determined until the end of the scanpath. The number of fixations is fixed. Visual part The visual part of the model is itself divided into 2 parts. The first part is a very simple retinal filter which provide a good fit to acuity limitations in the visual human field (Zelinsky, Zhang, Yu, Chen, & Samaras, 2006; Geisler & Perry, 2002) . The filtering output has a maximal resolution in areas near the fixation point, and the resolution decreases with eccen- tricity. More practically, locations close to the current fixa- tion will have a strong visual weight, and those further away a low weight. The second part of this process concerns visual stimuli. As in many models of visual attention (Itti & Koch, 2001) we also take into account this information in a pseudo saliency map. In the first experiment, this information is the