An active vision computational model of visual search for human-computer interaction

Visual search is an important part of human-computer interaction (HCI). The visual search processes that people use have a substantial effect on the time expended and likelihood of finding the information they seek. This dissertation investigates visual search through experiments and computational cognitive modeling. Computational cognitive modeling is a powerful methodology that uses computer simulation to capture, assert, record, and replay plausible sets of interactions among the many human processes at work during visual search. This dissertation aims to provide a cognitive model of visual search that can be utilized by predictive interface analysis tools and to do so in a manner consistent with a comprehensive theory of human visual processing, namely active vision. The model accounts for the four questions of active vision, the answers to which are important to both practitioners and researchers in HCI: What can be perceived in a fixation? When do the eyes move? Where do the eyes move? What information is integrated between eye movements? This dissertation presents a principled progression of the development of a computational model of active vision. Three experiments were conducted that investigate the effects of visual layout properties: density, color, and word meaning. The experimental results provide a better understanding of how these factors affect human-computer visual interaction. Three sets of data, two from the experiments reported here, were accurately modeled in the EPIC (Executive Process-Interactive Control) cognitive architecture. This work extends the practice of computational cognitive modeling by (a) informing the process of developing computational models through the use of eye movement data and (b) providing the first detailed instantiation of the theory of active vision in a computational framework. This instantiation allows us to better understand (a) the effects and interactions of visual search processes and (b) how these visual search processes can be used computationally to predict people's visual search behavior. This research ultimately benefits HCI by giving researchers and practitioners a better understanding of how users visually interact with computers and provides a foundation for tools to predict that interaction. This dissertation includes-both previously published and co-authored material.

[1]  Michael Wilson,et al.  MRC psycholinguistic database: Machine-usable dictionary, version 2.00 , 1988 .

[2]  Andrew Howes,et al.  Good Enough But I'll Just Check: Web-page Search as Attentional Refocusing , 2004, ICCM.

[3]  Michael D. Byrne,et al.  Modeling the Visual Search of Displays: A Revised ACT-R Model of Icon Search Based on Eye-Tracking Data , 2006, Hum. Comput. Interact..

[4]  Anthony J. Hornof,et al.  A minimal model for predicting visual search in human-computer interaction , 2007, CHI.

[5]  Anthony J. Hornof,et al.  Visual search and mouse-pointing in labeled versus unlabeled two-dimensional visual hierarchies , 2001, TCHI.

[6]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[7]  Jakob Nielsen,et al.  Top ten mistakes in web design , 1996 .

[8]  A. Newell.  Unified Theories of Cognition , 1990 .

[9]  J. Wolfe,et al.  What attributes guide the deployment of visual attention and how do they do it? , 2004, Nature Reviews Neuroscience.

[10]  David E. Kieras,et al.  An Overview of the EPIC Architecture for Cognition and Performance With Application to Human-Computer Interaction , 1997, Hum. Comput. Interact..

[11]  Anthony J. Hornof,et al.  Link colors guide a search , 2004, CHI EA '04.

[12]  Min-Shik Kim,et al.  The role of spatial working memory in visual search efficiency , 2004, Psychonomic bulletin & review.

[13]  A. Treisman.  Visual Attention and the Perception of Features and Objects , 1994 .

[14]  Anthony J. Hornof,et al.  The effects of semantic grouping on visual search , 2008, CHI Extended Abstracts.

[15]  Marc Pomplun,et al.  Guidance of eye movements during conjunctive visual search: the distractor-ratio effect. , 2003, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[16]  David E. Kieras,et al.  Predictive engineering models based on the EPIC architecture for a multimodal high-performance human-computer interaction task , 1997, TCHI.

[17]  G. Logan,et al.  On the Use of a Concurrent Memory Load to Measure Attention and Automaticity , 1979 .

[18]  M. D’Esposito.  Working memory. , 2008, Handbook of clinical neurology.

[19]  Gordon D. Logan,et al.  Attention in Character-Classification Tasks : Evidence for the Automaticity of Component Stages , 1978 .

[20]  Anthony J. Hornof,et al.  Strategy Shifts in Mixed Density Search , 2004 .

[21]  Carolyn Snyder,et al.  Web Site Usability: A Designer's Guide , 1997 .

[22]  Jeremy M. Wolfe,et al.  Guided Search 3.0 , 1997 .

[23]  Stuart K. Card,et al.  User perceptual mechanisms in the search of computer command menus , 1982, CHI '82.

[24]  Dario D. Salvucci.  An integrated model of eye movements and visual encoding , 2001, Cognitive Systems Research.

[25]  R. Näsänen,et al.  Eye movements in the visual search of word lists , 2002, Vision Research.

[26]  R. Logie.  Visuo-spatial Working Memory , 1994 .

[27]  Kevin Mullet,et al.  Designing Visual Interfaces: Communication Oriented Techniques , 1994 .

[28]  J. Findlay,et al.  Eye guidance and visual search , 1998 .

[29]  R. E. Christ.  Review and Analysis of Color Coding Research for Visual Displays , 1975 .

[30]  R. Klein,et al.  On the manifestations of memory in visual search. , 2000, Spatial vision.

[31]  W. Chase,et al.  Visual information processing. , 1974 .

[32]  Stephen J. Boies,et al.  Components of attention. , 1971 .

[33]  David E. Kieras,et al.  Visual Availability and Fixation Memory in Modeling Visual Search using the EPIC Architecture , 2006 .

[34]  Anthony J. Hornof,et al.  Explaining Eye Movements in the Visual Search of Varying Density Layouts , 2004, ICCM.

[35]  G. Woodman,et al.  Visual Search Remains Efficient when Visual Working Memory is Full , 2001, Psychological science.

[36]  Benjamin L. Somberg,et al.  A comparison of rule-based and positionally constant arrangements of computer menu items , 1986, CHI '87.

[37]  Kenneth R. Koedinger,et al.  Predictive human performance modeling made easy , 2004, CHI.

[38]  Eyal M. Reingold,et al.  Area activation: a computational model of saccadic selectivity in visual search , 2003, Cogn. Sci..

[39]  J. Findlay.  Saccade Target Selection During Visual Search , 1997, Vision Research.

[40]  C. J. Erkelens,et al.  Control of fixation duration in a simple search task , 1996, Perception & psychophysics.

[41]  Trey Hedden,et al.  Category norms as a function of culture and age: comparisons of item responses to 105 categories by american and chinese adults. , 2004, Psychology and aging.

[42]  N. Mackworth,et al.  The gaze selects informative details within pictures , 1967 .

[43]  G. W. Kootstra.  Bottom-up and Top-down Influences on Eye Movements , 2006 .

[44]  R. Klein,et al.  Inhibition of Return is a Foraging Facilitator in Visual Search , 1999 .

[45]  John R Anderson,et al.  An integrated theory of the mind. , 2004, Psychological review.

[46]  John R. Anderson,et al.  ACT-R: A Theory of Higher Level Cognition and Its Relation to Visual Attention , 1997, Hum. Comput. Interact..

[47]  Anthony J. Hornof,et al.  Local Density Guides Visual Search: Sparse Groups are First and Faster , 2004 .

[48]  Gerald L. Lohse,et al.  A Cognitive Model for Understanding Graphical Perception , 1993, Hum. Comput. Interact..

[49]  Andrew Howes,et al.  Strategies for Guiding Interactive Search: An Empirical Investigation Into the Consequences of Label Relevance for Assessment and Selection , 2008, Hum. Comput. Interact..

[50]  S. Hochstein,et al.  How Serial is Serial Processing in Vision? , 1989, Perception.

[51]  A. Newell.  You can't play 20 questions with nature and win : projective comments on the papers of this symposium , 1973 .

[52]  Waldemar Karwowski,et al.  Top Ten Mistakes in Web Design , 2006 .

[53]  Richard L. Lewis,et al.  Supporting efficient development of cognitive models at multiple skill levels: exploring recent advances in constraint-based modeling , 2005, CHI.

[54]  J M Wolfe,et al.  Search for multiple targets: Remember the targets, forget the search , 2001, Perception & psychophysics.

[55]  Edward K. Vogel,et al.  The capacity of visual working memory for features and conjunctions , 1997, Nature.

[56]  G. Sperling,et al.  Is there feature-based attentional selection in visual search? , 1996, Journal of experimental psychology. Human perception and performance.

[57]  David E. Kieras,et al.  Using GOMS for user interface design and evaluation: which technique? , 1996, TCHI.

[58]  Anthony J. Hornof,et al.  Cognitive Strategies for the Visual Search of Hierarchical Computer Displays , 2004, Hum. Comput. Interact..

[59]  Dario D. Salvucci.  Predicting the effects of in-car interface use on driver performance: an integrated model approach , 2001, Int. J. Hum. Comput. Stud..

[60]  Bonnie E. John,et al.  Towards a Tool for Predicting Goal-directed Exploratory Behavior , 2008 .

[61]  Alastair G. Gale,et al.  Visual search 2 , 1993 .

[62]  Bonnie E. John,et al.  Multipurpose prototypes for assessing user interfaces in pervasive computing systems , 2005, IEEE Pervasive Computing.

[63]  F. Wilkinson,et al.  Distractor Ratio and Grouping Processes in Visual Conjunction Search , 1992, Perception.

[64]  Richard P. Heitz,et al.  An automated version of the operation span task , 2005, Behavior research methods.

[65]  J. Wolfe,et al.  Guided Search 2.0 A revised model of visual search , 1994, Psychonomic bulletin & review.

[66]  R. Snowden,et al.  Can one pay attention to a particular color? , 1999, Perception & psychophysics.

[67]  J. Findlay,et al.  Active Vision: The Psychology of Looking and Seeing , 2003 .

[68]  G. Underwood.  Eye guidance in reading and scene perception , 1998 .

[69]  John Anderson,et al.  An integrated theory of prospective time interval estimation: the role of cognition, attention, and learning. , 2007, Psychological review.

[70]  Michael D. Byrne,et al.  ACT-R/PM and menu selection: applying a cognitive architecture to HCI , 2001, Int. J. Hum. Comput. Stud..

[71]  A. Hornof,et al.  Towards a Flexible, Reusable Model for Predicting Eye Movements During Visual Search of Text , 2006 .

[72]  J. L. Barbur,et al.  Eye movements and search performance , 1993 .

[73]  V. Lakshminarayanan,et al.  Basic and Clinical Applications of Vision Science , 1997, Documenta Ophthalmologica Proceedings Series.

[74]  Anthony J. Hornof,et al.  Cognitive strategies and eye movements for searching hierarchical computer displays , 2003, CHI '03.

[75]  David E. Kieras,et al.  The GOMS family of user interface analysis techniques: comparison and contrast , 1996, TCHI.

[76]  G. Perlman.  Making the right choices with menus , 1987 .

[77]  J. Wolfe,et al.  A Model of Visual Search Catches Up With Jay Enoch 40 Years Later , 1999 .

[78]  Wai-Tat Fu,et al.  SNIF-ACT: A Cognitive Model of User Navigation on the World Wide Web , 2007, Hum. Comput. Interact..

[79]  B. C. Motter,et al.  The guidance of eye movements during active visual search , 1998, Vision Research.

[80]  J. H. Bertera,et al.  Eye movements and the span of the effective stimulus in visual search , 2000, Perception & psychophysics.

[81]  C Casco,et al.  Spatial interactions in simple and combined-feature visual search. , 1999, Spatial vision.

[82]  M. Pomplun,et al.  Distractor Ratio Influences Patterns of Eye Movements during Visual Search , 2000, Perception.

[83]  Tim Halverson,et al.  Cleaning up systematic error in eye-tracking data by using required fixation locations , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.