Attention and Visual Memory in Visualization and Computer Graphics

A fundamental goal of visualization is to produce images of data that support visual analysis, exploration, and discovery of novel insights. An important consideration during visualization design is the role of human visual perception. How we "see” details in an image can directly impact a viewer's efficiency and effectiveness. This paper surveys research on attention and visual perception, with a specific focus on results that have direct relevance to visualization and visual analytics. We discuss theories of low-level visual perception, then show how these findings form a foundation for more recent work on visual memory and visual attention. We conclude with a brief overview of how knowledge of visual attention and visual memory is being applied in visualization and graphics. We also discuss how challenges in visualization are motivating research in psychophysics.

[1]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[2]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[3]  J. Lund,et al.  Compulsory averaging of crowded orientation signals in human vision , 2001, Nature Neuroscience.

[4]  O. Velev,et al.  Materials: A class of porous metallic nanostructures , 1999, Nature.

[5]  Ken Nakayama,et al.  Serial and parallel processing of visual feature conjunctions , 1986, Nature.

[6]  A. Kingstone,et al.  Get real! Resolving the debate about equivalent social stimuli , 2009 .

[7]  A. Rosenfeld,et al.  A Theory of Textural Segmentation , 1983 .

[8]  K. Rayner.  Eye movements in reading and information processing. , 1978, Psychological bulletin.

[9]  Ronald A. Rensink,et al.  Change blindness: past, present, and future , 2005, Trends in Cognitive Sciences.

[10]  Ronald A. Rensink,et al.  Influence of scene-based properties on visual search. , 1990, Science.

[11]  S. Yantis,et al.  Visual attention: control, representation, and time course. , 1997, Annual review of psychology.

[12]  Harold Pashler,et al.  A Boolean map theory of visual attention. , 2007, Psychological review.

[13]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[14]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[15]  Ronald A. Rensink,et al.  Consequences of display changes during interrupted visual search: Rapid resumption is target specific , 2007, Perception & psychophysics.

[16]  L. Stark,et al.  Scanpaths in saccadic eye movements while viewing and recognizing patterns. , 1971, Vision research.

[17]  Andrei Gorea,et al.  Two carriers for motion perception: Color and luminance , 1991, Vision Research.

[18]  Christopher G. Healey,et al.  Choosing effective colours for data visualization , 1996, Proceedings of Seventh Annual IEEE Visualization '96.

[19]  L. Stark,et al.  Scanpaths in Eye Movements during Pattern Perception , 1971, Science.

[20]  D. Ariely.  Seeing Sets: Representation by Statistical Properties , 2001, Psychological science.

[21]  B Julesz,et al.  Experiments in the visual perception of texture. , 1975, Scientific American.

[22]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[23]  James T. Enns,et al.  Real-Time Multivariate Data Visualization Using Preattentive Processing , 1994 .

[24]  V. Gallese,et al.  Motion, emotion and empathy in esthetic experience , 2007, Trends in Cognitive Sciences.

[25]  Chih-Jen Wei,et al.  Characterizing the Limits of Human Visual Awareness , 2007 .

[26]  J. P. Thomas,et al.  A signal detection model predicts the effects of set size on visual search accuracy for feature, conjunction, triple conjunction, and disjunction displays , 2000, Perception & psychophysics.

[27]  Kimron Shapiro,et al.  Direct measurement of attentional dwell time in human vision , 1994, Nature.

[28]  D. A. Duce,et al.  Visualization in Scientific Computing , 1994, Focus on Computer Graphics.

[29]  A. Kingstone,et al.  Saliency does not account for fixations to eyes within social scenes , 2009, Vision Research.

[30]  A. Treisman,et al.  Illusory words: the roles of attention and of top-down constraints in conjoining letters to form words. , 1986, Journal of experimental psychology. Human perception and performance.

[31]  A. Nagy,et al.  Critical color differences determined with a visual search task. , 1990, Journal of the Optical Society of America. A, Optics and image science.

[32]  Anne Treisman,et al.  Preattentive processing in vision , 1985, Computer Vision Graphics and Image Processing.

[33]  J. Wolfe,et al.  Guided Search 2.0 A revised model of visual search , 1994, Psychonomic bulletin & review.

[34]  Patrick Cavanagh,et al.  Perceiving Illumination Inconsistencies in Scenes , 2005, Perception.

[35]  K Nakayama,et al.  Visual attention to surfaces in three-dimensional space. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[36]  James T. Enns,et al.  Engaging viewers through nonphotorealistic visualizations , 2007, NPAR '07.

[37]  Manuel Menezes de Oliveira Neto,et al.  An Efficient Naturalness-Preserving Image-Recoloring Method for Dichromats , 2008, IEEE Transactions on Visualization and Computer Graphics.

[38]  J. Townsend.  Serial vs. Parallel Processing: Sometimes They Look like Tweedledum and Tweedledee but they can (and Should) be Distinguished , 1990 .

[39]  B. Tatler,et al.  The Moving Tablet Of The Eye , 2005 .

[40]  M. Land,et al.  The Roles of Vision and Eye Movements in the Control of Activities of Daily Living , 1998, Perception.

[41]  Raymond Klein,et al.  Inhibition of return , 2000, Trends in Cognitive Sciences.

[42]  J. Duncan.  Boundary Conditions on Parallel Processing in Human Vision , 1989, Perception.

[43]  A Treisman,et al.  Feature analysis in early vision: evidence from search asymmetries. , 1988, Psychological review.

[44]  P. Winkielman,et al.  PSYCHOLOGICAL SCIENCE Research Article Prototypes Are Attractive Because They Are Easy on the Mind , 2022 .

[45]  A. Treisman,et al.  Representation of statistical properties , 2003, Vision Research.

[46]  Nancy Kanwisher,et al.  fMRI evidence for objects as the units of attentional selection , 1999, Nature.

[47]  P. McLeod,et al.  Motion coherence and conjunction search: Implications for guided search theory , 1992, Perception & psychophysics.

[48]  J. W. Gebhard,et al.  Difference-Limens for Photic Intermittence* , 1955 .

[49]  B. Julesz,et al.  Human factors and behavioral science: Textons, the fundamental elements in preattentive vision and perception of textures , 1983, The Bell System Technical Journal.

[50]  P. Sterling,et al.  How Much the Eye Tells the Brain , 2006, Current Biology.

[51]  William Knight,et al.  Using visual texture for information display , 1995, TOGS.

[52]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[53]  J. Enns,et al.  What's next? New evidence for prediction in human vision , 2008, Trends in Cognitive Sciences.

[54]  G H MOWBRAY,et al.  Differential sensitivity of the eye to intermittent white light. , 1955, Science.

[55]  Michelle R. Greene,et al.  Visual search in scenes involves selective and nonselective pathways , 2011, Trends in Cognitive Sciences.

[56]  Bela Julesz,et al.  A theory of preattentive texture discrimination based on first-order statistics of textons , 2004, Biological Cybernetics.

[57]  R. Snowden,et al.  Texture segregation and visual search: a comparison of the effects of random variations along irrelevant dimensions. , 1998, Journal of experimental psychology. Human perception and performance.

[58]  James T. Enns,et al.  Target detection and localization in visual search: A dual systems perspective , 2003, Perception & psychophysics.

[59]  Ronald A. Rensink,et al.  Rapid Resumption of Interrupted Visual Search , 2005, Psychological science.

[60]  J. Wolfe,et al.  Post-attentive vision , 2001 .

[61]  E. Gibson.  Principles of Perceptual Learning and Development , 1969 .

[62]  T. Callaghan.  Interference and dominance in texture segregation: Hue, geometric form, and line orientation , 1989, Perception & psychophysics.

[63]  B. Julesz.  Textons, the elements of texture perception, and their interactions , 1981, Nature.

[64]  James T. Enns,et al.  Large Datasets at a Glance: Combining Textures and Colors in Scientific Visualization , 1999, IEEE Trans. Vis. Comput. Graph..

[65]  D. Kahneman,et al.  Attention and Effort , 1973 .

[66]  Ann McNamara,et al.  Subtle gaze direction , 2009, TOGS.

[67]  K L Shapiro,et al.  Temporary suppression of visual processing in an RSVP task: an attentional blink? . , 1992, Journal of experimental psychology. Human perception and performance.

[68]  C. Chabris,et al.  Gorillas in Our Midst: Sustained Inattentional Blindness for Dynamic Events , 1999, Perception.

[69]  O. Mimura.  [Eye movements]. , 1992, Nippon Ganka Gakkai zasshi.

[70]  Daniel J. Simons,et al.  Current Approaches to Change Blindness , 2000 .

[71]  B Julesz,et al.  "Where" and "what" in vision. , 1985, Science.

[72]  Alan Chalmers,et al.  Detail to Attention: Exploiting Visual Tasks for Selective Rendering , 2003, Rendering Techniques.

[73]  Susan L. Franzel,et al.  Guided search: an alternative to the feature integration model for visual search. , 1989, Journal of experimental psychology. Human perception and performance.

[74]  S. Mateeff,et al.  The time it takes to detect changes in speed and direction of visual motion , 1998, Vision Research.

[75]  Colin Ware,et al.  Color sequences for univariate maps: theory, experiments and principles , 1988, IEEE Computer Graphics and Applications.

[76]  Victoria Interrante,et al.  Harnessing natural textures for multivariate visualization , 2000, IEEE Computer Graphics and Applications.

[77]  Ronald A. Rensink,et al.  TO SEE OR NOT TO SEE: The Need for Attention to Perceive Changes in Scenes , 1997 .

[78]  Susan L. Franzel,et al.  Binocularity and visual search , 1988, Perception & psychophysics.

[79]  Donald P. Greenberg,et al.  Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments , 2001, TOGS.

[80]  James T. Enns,et al.  Building perceptual textures to visualize multidimensional datasets , 1998 .

[81]  Timothy F. Brady,et al.  The contents of perceptual hypotheses: Evidence from rapid resumption of interrupted visual search , 2009, Attention, perception & psychophysics.

[82]  B. Julesz,et al.  Detection versus Discrimination of Visual Orientation , 1984, Perception.

[83]  R. Sekuler,et al.  Motion processing in peripheral vision: Reaction time and perceived velocity , 1982, Vision Research.

[84]  D. Simons,et al.  Failure to detect changes to attended objects in motion pictures , 1997 .

[85]  Zenzi M. Griffin,et al.  Why Look? Reasons for Eye Movements Related to Language Production. , 2004 .

[86]  K. Jellinger.  The Moving Tablet of the Eye: The Origins of Modern Eye Movement Research , 2006 .

[87]  D. Ballard,et al.  What you see is what you need. , 2003, Journal of vision.

[88]  J E Hoffman,et al.  A two-stage model of visual search , 1979, Perception & psychophysics.

[89]  J. R. Pomerantz,et al.  Emergent features, attention, and perceptual glue in visual form perception. , 1989, Journal of experimental psychology. Human perception and performance.

[90]  Ronald A. Rensink.  Seeing, sensing, and scrutinizing , 2000, Vision Research.

[91]  J. Duncan.  The locus of interference in the perception of simultaneous stimuli. , 1980, Psychological review.

[92]  J. Duncan,et al.  Visual search and stimulus similarity. , 1989, Psychological review.

[93]  D L Sheinberg,et al.  PSYCHOLOGICAL SCIENCE Research Article ATTENTIONAL ENHANCEMENT OPPOSITE A PERIPHERAL FLASH REVEALED USING CHANGE BLINDNESS , 2022 .

[94]  Jason M Haberman,et al.  Seeing the mean: ensemble coding for sets of faces. , 2009, Journal of experimental psychology. Human perception and performance.

[95]  James T. Enns,et al.  Harnessing Preattentive Processes for Multivariate Data Visualization , 1992 .

[96]  D. Simons.  In Sight, Out of Mind: When Object Representations Fail , 1996 .

[97]  Z. Pylyshyn,et al.  Why are small and large numbers enumerated differently? A limited-capacity preattentive stage in vision. , 1994, Psychological review.

[98]  Christopher G. Healey,et al.  Visualizing multidimensional query results using animation , 2008, Electronic Imaging.

[99]  Frederick A.A. Kingdom,et al.  Lightness, brightness and transparency: A quarter century of new ideas, captivating demonstrations and unrelenting controversy , 2011, Vision Research.

[100]  A. Treisman.  Search, similarity, and integration of features between and within dimensions. , 1991, Journal of experimental psychology. Human perception and performance.

[101]  W. Cowan,et al.  Visual search for colour targets that are or are not linearly separable from distractors , 1996, Vision Research.

[102]  Anne Treisman,et al.  Monitoring and storage of irrelevant messages in selective attention , 1964 .

[103]  Christopher G. Healey,et al.  Attribute preserving dataset simplification , 2001, Proceedings Visualization, 2001. VIS '01..

[104]  Colin Ware,et al.  Filtering and Integrating Visual Information with Motion , 2001 .

[105]  J. Deutsch.  Perception and Communication , 1958, Nature.

[106]  D. Broadbent,et al.  From detection to identification: Response to multiple targets in rapid serial visual presentation , 1987, Perception & psychophysics.

[107]  Lucy T. Nowell,et al.  Change blindness in information visualization: a case study , 2001, IEEE Symposium on Information Visualization, 2001. INFOVIS 2001..

[108]  C. A. Garabedian.  Birkhoff on Aesthetic Measure , 1934 .

[109]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[110]  Douglas DeCarlo,et al.  Visual interest and NPR: an evaluation and manifesto , 2004, NPAR '04.

[111]  A. Treisman,et al.  Statistical processing: computing the average size in perceptual groups , 2005, Vision Research.

[112]  M. Chun,et al.  Contextual cueing of visual attention , 2022 .

[113]  B. Julesz.  A brief outline of the texton theory of human vision , 1984, Trends in Neurosciences.

[114]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[115]  Ronald A. Rensink,et al.  Sensitivity To Three-Dimensional Orientation in Visual Search , 1990 .

[116]  David S. Ebert,et al.  Non-photorealistic volume rendering using stippling techniques , 2002, IEEE Visualization, 2002. VIS 2002..

[117]  Edward A. Vessel,et al.  Perceptual Pleasure and the Brain , 2006 .

[118]  Walter F. Bischof,et al.  Attentional switching in spatial and nonspatial domains : Evidence from the attentional blink , 1999 .

[119]  James T. Enns,et al.  Perceptually based brush strokes for nonphotorealistic visualization , 2004, TOGS.

[120]  Alan Chalmers,et al.  Varying rendering fidelity by exploiting human change blindness , 2003, GRAPHITE '03.

[121]  T. Callaghan.  Dimensional interaction of hue and brightness in preattentive field segregation , 1984, Perception & psychophysics.

[122]  A. Treisman,et al.  Binding in short-term visual memory. , 2002, Journal of experimental psychology. General.

[123]  Monica S. Castelhano,et al.  Eye Movements and Visual Memory for Scenes , 2005 .

[124]  G W Humphreys,et al.  Visual search for targets defined by combinations of color, shape, and size: An examination of the task constraints on feature and conjunction searches , 1987, Perception & psychophysics.

[125]  J. Wolfe,et al.  The role of categorization in visual search for orientation. , 1992, Journal of experimental psychology. Human perception and performance.

[126]  Christopher G. Healey,et al.  Visualizing data with motion , 2005, VIS 05. IEEE Visualization, 2005..

[127]  Benjamin Y Hayden,et al.  Economic principles motivating social attention in humans , 2007, Proceedings of the Royal Society B: Biological Sciences.

[128]  Fernanda Ferreira,et al.  Scene Perception for Psycholinguists. , 2004 .

[129]  Michael D'Zmura,et al.  Color in visual search , 1991, Vision Research.

[130]  J. Wolfe,et al.  Limitations on the Parallel Guidance of Visual Search : Color x Color and Orientation x Orientation Conjuctions , 2004 .

[131]  M. Chun,et al.  Contextual Cueing: Implicit Learning and Memory of Visual Context Guides Spatial Attention , 1998, Cognitive Psychology.

[132]  James T. Enns,et al.  Visualizing real-time multivariate data using preattentive processing , 1995, TOMC.

[133]  D. Berlyne,et al.  Aesthetics and Psychobiology , 1975 .

[134]  L. Stark,et al.  Evidence for a global scanpath strategy in viewing abstract compared with realistic images , 1995, Neuropsychologia.

[135]  M. Posner,et al.  Components of visual orienting , 1984 .

[136]  M. Posner,et al.  The attention system of the human brain. , 1990, Annual review of neuroscience.

[137]  B. Julesz,et al.  A unified approach to the perception of motion, stereo, and static-flow patterns , 1995 .

[138]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[139]  J. Pokorny.  Foundations of Cyclopean Perception , 1972 .

[140]  P. Cavanagh,et al.  Attentional resolution , 1997, Trends in Cognitive Sciences.

[141]  J. Grimes.  On the failure to detect changes in scenes across saccades. , 1996 .