Gaze-Contingent Multiresolutional Displays: An Integrative Review

Gaze-contingent multiresolutional displays (GCMRDs) center high-resolution information on the user's gaze position, matching the user's area of interest (AOI). Image resolution and details outside the AOI are reduced, lowering the requirements for processing resources and transmission bandwidth in demanding display and imaging applications. This review provides a general framework within which GCMRD research can be integrated, evaluated, and guided. GCMRDs (or “moving windows”) are analyzed in terms of (a) the nature of their images (i.e., “multiresolution,” “variable resolution,” “space variant,” or “level of detail”), and (b) the movement of the AOI (i.e., “gaze contingent,” “foveated,” or “eye slaved”). We also synthesize the known human factors research on GCMRDs and point out important questions for future research and development. Actual or potential applications of this research include flight, medical, and driving simulators; virtual reality; remote piloting and teleoperation; infrared and indirect vision; image transmission and retrieval; telemedicine; video teleconferencing; and artificial vision systems.

[1]  B. Dierickx,et al.  Image-based personal communication using an innovative space-variant CMOS sensor , 1996, Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN'96 TSUKUBA.

[2]  Nicholas M. Dalton,et al.  Photo Based Image Generator , 1989, Defense, Security, and Sensing.

[3]  Eyal M. Reingold,et al.  Saccadic Inhibition in Voluntary and Reflexive Saccades , 2002, Journal of Cognitive Neuroscience.

[4]  Alex Waibel,et al.  A model-based gaze tracking system , 1996, Proceedings IEEE International Joint Symposia on Intelligence and Systems.

[5]  Wa James Tam,et al.  Processing image sequences based on eye movements , 1994, Electronic Imaging.

[6]  Benjamin Watson,et al.  Managing level of detail through peripheral degradation: effects on search performance with a head-mounted display , 1997, TCHI.

[7]  D. Roenker,et al.  Age and visual search: expanding the useful field of view. , 1988, Journal of the Optical Society of America. A, Optics and image science.

[8]  F. O. Huck,et al.  Visual Information Processing , 1992 .

[9]  K. I. Kim,et al.  Collision avoidance using artificial retina sensor in ALV , 1995, Proceedings of the Intelligent Vehicles '95. Symposium.

[10]  J. Rensbergen,et al.  Perception and cognition : advances in eye movement research , 1995 .

[11]  R. F. Hess,et al.  The contrast sensitivity gradient across the human visual field: With emphasis on the low spatial frequency range , 1989, Vision Research.

[12]  S. Shioiri,et al.  Useful Resolution for Picture Perception as a Function of Eccentricity , 1989, Perception.

[13]  Bernice E. Rogowitz,et al.  Human Vision and Electronic Imaging II , 1997 .

[14]  Lester C. Loschky,et al.  User performance with gaze contingent multiresolutional displays , 2000, ETRA.

[15]  J. H. Bertera,et al.  Eye movements and the span of the effective stimulus in visual search , 2000, Perception & psychophysics.

[16]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[17]  Lester C. Loschky,et al.  Perception onset time during fixations in free viewing , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[18]  P. Bennett,et al.  Effects of aging on the useful field of view. , 2000, Experimental aging research.

[19]  Paul C. Chevrette,et al.  Wide-area-coverage infrared surveillance system , 1996, Defense, Security, and Sensing.

[20]  Frank L. Kooi.  Binocular configurations of a night-flight head-mounted display , 1993 .

[21]  Douglas E. McGovern,et al.  Experience and results in teleoperation of land vehicles , 1991 .

[22]  Kathryn T. Spoehr,et al.  Visual Information Processing , 1982 .

[23]  John R. Quick.  System requirements for a high-gain dome-display surface , 1990, Other Conferences.

[24]  Romeo Chua,et al.  On-line control of rapid aiming movements: Unexpected target perturbations and movement kinematics. , 1998 .

[25]  Martin Reddy,et al.  Specification and evaluation of level of detail selection criteria , 1998, Virtual Reality.

[26]  Norimichi Tsumura,et al.  Image compression and decompression based on gazing area , 1996, Electronic Imaging.

[27]  W. P. Leavy,et al.  Closing the Gap between Aircraft and Simulator Training with Limited Field-of-View Visual Systems, , 1983 .

[28]  J. Robson,et al.  Application of fourier analysis to the visibility of gratings , 1968, The Journal of physiology.

[29]  Bruce H. McCormick,et al.  Gaze-contingent video resolution degradation , 1998, Electronic Imaging.

[30]  W. Helsen,et al.  Temporal and spatial coupling of point of gaze and hand movements in aiming. , 1998, Journal of motor behavior.

[31]  A. Bradley,et al.  Characterization of spatial aliasing and contrast sensitivity in peripheral vision , 1996, Vision Research.

[32]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[33]  Timothy F. Gee,et al.  Remote driving with reduced bandwidth communication , 1992 .

[34]  N. Charness,et al.  Visual Span in Expert Chess Players: Evidence From Eye Movements , 2001, Psychological science.

[35]  Giulio Sandini,et al.  Visuo-inertial stabilization in space-variant binocular systems , 2000, Robotics Auton. Syst..

[36]  B. Kappe,et al.  HEAD-SLAVED IMAGES FOR LOW-COST DRIVING SIMULATORS AND UNMANNED GROUND VEHICLES , 1997 .

[37]  Marc Levoy,et al.  Gaze-directed volume rendering , 1990, I3D '90.

[38]  Wilson S. Geisler,et al.  Implementation of a foveated image coding system for image bandwidth reduction , 1996, Electronic Imaging.

[39]  Gregory K. Tharp,et al.  A helmet mounted display to adapt the telerobotic environment to human vision , 1990 .

[40]  John Pretlove,et al.  Design of a high-performance telepresence system incorporating an active vision system for enhanced visual perception of remote environments , 1995, Other Conferences.

[41]  Richard A. Bolt,et al.  The human interface: Where people and computers meet , 1984 .

[42]  Eugenio Culurciello,et al.  Evaluating variable resolution displays with visual search: task performance and eye movements , 2000, ETRA.

[43]  I. S. Mackenzie,et al.  Virtual Environments and Advanced Interface Design , 1995 .

[44]  B Séré,et al.  Nonhomogeneous Resolution of Images of Natural Scenes , 2000, Perception.

[45]  Anthony J. Maeder,et al.  Limiting human perception for image sequences , 1996, Electronic Imaging.

[46]  Y Y Zeevi,et al.  Visual assessment of variable-resolution imagery. , 1995, Journal of the Optical Society of America. A, Optics, image science, and vision.

[47]  J. E. Korteling,et al.  Effects of Head-Slaved and Peripheral Displays on Lane-Keeping Performance and Spatial Orientation , 1999, Hum. Factors.

[48]  Shumin Zhai,et al.  Hand eye coordination patterns in target selection , 2000, ETRA.

[49]  Gary L. Serfoss,et al.  Effects of Area-of-Interest Display Characteristics of Visual Search Performance and Head Movements in Simulated Low-Level Flight , 1993 .

[50]  Zhou Wang,et al.  Embedded foveation image coding , 2001, IEEE Trans. Image Process..

[51]  Antal K. Bejczy,et al.  Telerobotics surgery in a transatlantic experiment: application in laparoscopy , 1993, Other Conferences.

[52]  Gordon W. Roberts,et al.  A log-polar image sensor fabricated in a standard 1.2-/spl mu/m ASIC CMOS process , 1997 .

[53]  Carl F. R. Weiman.  Video compression by matching human perceptual channels , 1994, Defense, Security, and Sensing.

[54]  G. Sperling,et al.  The dimensionality of texture-defined motion: a single channel theory , 1993, Vision Research.

[55]  R. E. Barrette.  Flight Simulator Visual Systems - An Overview , 1986 .

[56]  Thierry Pun,et al.  Integration of bottom-up and top-down cues for visual attention using non-linear relaxation , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Woodrow Barfield,et al.  Virtual environments and advanced interface design , 1995 .

[58]  Martin Reddy,et al.  Perceptually modulated level of detail for virtual environments , 1997 .

[59]  Ronald G Hughes,et al.  Tactical Ground Attack: On the Transfer of Training from Flight Simulator to Operational Red Flag Range Exercise , 1982 .

[60]  Regan Ec,et al.  The frequency of occurrence and severity of side-effects of immersion virtual reality. , 1994 .

[61]  P. O. Bishop,et al.  Spatial vision. , 1971, Annual review of psychology.

[62]  Lester C. Loschky,et al.  Investigating spatial vision and dynamic attentional selection using a gaze-contingent multiresolutional display. , 2002 .

[63]  Martin Reddy,et al.  A Perceptual Framework for Optimising Visual Detail in Virtual Environments , 1999 .

[64]  Alexander Zelinsky,et al.  An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[65]  E. C. Regan,et al.  The frequency of occurrence and severity of side-effects of immersion virtual reality. , 1994, Aviation, space, and environmental medicine.

[66]  Jerry D. Gibson,et al.  Handbook of Image and Video Processing , 2000 .

[67]  Robert J. K. Jacob,et al.  Eye tracking in advanced interface design , 1995 .

[68]  John G. Casali,et al.  Effects of Visual Display and Motion System Delays on Operator Performance and Uneasiness in a Driving Simulator , 1987 .

[69]  L. Young,et al.  Survey of eye movement recording methods , 1975 .

[70]  Wilson S. Geisler,et al.  Variable‐Resolution Displays for Visual Communication and Simulation , 1999 .

[71]  Eric L. Schwartz,et al.  Design considerations for a space-variant visual sensor with complex-logarithmic geometry , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[72]  Howell O. Istance,et al.  Keeping an Eye on your Interface: Potential for Eye-Based Control of Graphical User Interfaces (GUI's) , 1994, BCS HCI.

[73]  Loran A. Haworth,et al.  Effect of field-of-view restriction on rotocraft pilot workload and performance: preliminary results , 1995, Defense, Security, and Sensing.

[74]  Seiji Inokuchi,et al.  A new anthropomorphic retina-like visual sensor , 1994, Proceedings of the 12th IAPR International Conference on Pattern Recognition, Vol. 2 - Conference B: Computer Vision & Image Processing. (Cat. No.94CH3440-5).

[75]  Jyrki Rovamo,et al.  Detection of chromatic deviations from white across the human visual field , 1991, Vision Research.

[76]  Lester C. Loschky,et al.  Investigating *Perception and Eye Movement Control in Natural Scenes Using Gaze -Contingent Multi -Resolutional Displays , 2002 .

[77]  M. R. Haswell.  Visual systems developments , 1986 .

[78]  Peter J. Thomas,et al.  Medical image databases: the ICOS project , 1999 .

[79]  Rynson W. H. Lau,et al.  An Adaptive Multiresolution Method for Progressive Model Transmission , 2001, Presence: Teleoperators & Virtual Environments.

[80]  Gerrit T. Viljoen.  Comparative study of target acquisition performance between an eye-slaved helmet display and unaided human vision , 1998, Defense, Security, and Sensing.

[81]  L N Thibos,et al.  Acuity perimetry and the sampling theory of visual resolution. , 1998, Optometry and vision science : official publication of the American Academy of Optometry.

[82]  Wa James Tam,et al.  Static and dynamic spatial resolution in image coding: an investigation of eye movements , 1991, Electronic Imaging.

[83]  P M van Diepen,et al.  Scene Exploration with Fourier-Filtered Peripheral Information , 1998, Perception.

[84]  S. Ellis.  Pictorial communication in virtual and real environments , 1991 .

[85]  Gordon W. Roberts,et al.  A foveated image sensor in standard CMOS technology , 1995, Proceedings of the IEEE 1995 Custom Integrated Circuits Conference.

[86]  Yongmin Kim,et al.  Multimedia systems for telemedicine and their communications requirements , 1996 .

[87]  Jian Yang,et al.  Subjective Evaluation of Retinal-Dependent Image Degradations , 2001, PICS.

[88]  D. S. Wooding,et al.  Fixation Patterns Made during Brief Examination of Two-Dimensional Images , 1997, Perception.

[89]  John H. Reif,et al.  Optical design and analysis of a head-mounted display with a high-resolution insert , 1995, Optics & Photonics.

[90]  Lester C. Loschky,et al.  Saliency of peripheral targets in gaze-contingent multiresolutional displays , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[91]  Andrew T. Duchowski,et al.  EUROGRAPHICS 2001 / Jonathan C. Roberts Short Presentations Gaze-Contingent Level Of Detail Rendering , 2022 .

[92]  Mark S. Rolwes.  Design and flight testing of an electronic visibility system , 1990, Defense, Security, and Sensing.

[93]  Mark H. Draper,et al.  Effects of Image Scale and System Time Delay on Simulator Sickness within Head-Coupled Virtual Environments , 2001, Hum. Factors.

[94]  Effects of field of view on performance with head-mounted displays , 2000 .

[95]  M. Pomplun,et al.  Investigating the visual span in comparative search: the effects of task difficulty and divided attention , 2001, Cognition.

[96]  Lester C. Loschky,et al.  Investigating spatial vision and dynamic attentional selection using a gaze-contingent multiresolutional display. , 2002, Journal of experimental psychology. Applied.

[97]  Gordon H. Robinson,et al.  Dynamics of the Eye and Head during Movement between Displays: A Qualitative and Quantitative Guide for Designers , 1979 .

[98]  D. Burr,et al.  Changes in visual perception at the time of saccades , 2001, Trends in Neurosciences.

[99]  K S Berbaum.  Design Criteria for Reducing 'Popping' in Area-of-Interest Displays: Preliminary Experiments. , 1983 .

[100]  S. Kohn,et al.  Visual Field Information in Low-Altitude Visual Flight by Line-of-Sight Slaved Helmet-Mounted Displays , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[101]  J M Thompson,et al.  Human factors in telesurgery: effects of time delay and asynchrony in video and control feedback with local manipulative assistance. , 1999, Telemedicine journal : the official journal of the American Telemedicine Association.

[102]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[103]  Allen Tannenbaum,et al.  On the eye tracking problem: a challenge for robust control , 2000 .

[104]  Helder Araújo,et al.  Optical Normal Flow Estimation on Log-polar Images. A Solution for Real-Time Binocular Vision , 1997, Real Time Imaging.

[105]  Wilson S. Geisler,et al.  IEEE TRANSACTIONS ON SYSTEMS , MAN , AND CYBERNETICS — PART A : SYSTEMS AND HUMANS , 2009 .

[106]  R. A. Fisher,et al.  Progress Report on an Eye-Slaved Area-of-Interest Visual Display, , 1984 .

[107]  Hideyuki Tamura,et al.  Gaze-directed adaptive rendering for interacting with virtual space , 1996, Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium.

[108]  Louis B. Rosenberg.  The use of Virtual Fixtures to Enhance Operator Performance in Time Delayed Teleoperation. , 1993 .

[109]  W W Wierwille,et al.  Effects of Visual Display and Motion System Delays on Operator Performance and Uneasiness in a Driving Simulator , 1988, Human factors.

[110]  D. Guitton,et al.  Gaze control in humans: eye-head coordination during orienting movements to targets within and beyond the oculomotor range. , 1987, Journal of neurophysiology.

[111]  Ernst Niebur,et al.  Variable-Resolution Displays: A Theoretical, Practical, and Behavioral Evaluation , 2002, Hum. Factors.

[112]  Andrew T. Duchowski.  Acuity-matching resolution degradation through wavelet coefficient scaling , 2000, IEEE Trans. Image Process..

[113]  Alfred T Lee,et al.  Visual Scene Simulation Requirements for C-5A/C-141B Aerial Refueling Part Task Trainer. , 1983 .

[114]  Wilson S. Geisler,et al.  Real-time foveated multiresolution system for low-bandwidth video communication , 1998, Electronic Imaging.

[115]  Vladimir Pavlovic,et al.  Toward multimodal human-computer interface , 1998, Proc. IEEE.

[116]  John H. Reif,et al.  Design and applications of a high-resolution insert head-mounted-display , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[117]  G Sandini,et al.  New developments in low-bit rate videotelephony for people who are deaf. , 1997, Journal of speech, language, and hearing research : JSLHR.

[118]  Kenneth Zeger,et al.  Progressive image coding with spatially variable resolution , 1997, Proceedings of International Conference on Image Processing.

[119]  Giulio Sandini,et al.  A retina-like CMOS sensor and its applications , 2000, Proceedings of the 2000 IEEE Sensor Array and Multichannel Signal Processing Workshop. SAM 2000 (Cat. No.00EX410).

[120]  Carl F. R. Weiman,et al.  Video compression via log polar mapping , 1990, Defense, Security, and Sensing.

[121]  E. Peli,et al.  Discrimination of wide-field images as a test of a peripheral-vision model. , 2001, Journal of the Optical Society of America. A, Optics, image science, and vision.

[122]  Anup Basu,et al.  Enhancing videoconferencing using spatially varying sensing , 1998, IEEE Trans. Syst. Man Cybern. Part A.

[123]  M M Shabot,et al.  Telesurgery. Acceptability of compressed video for remote surgical proctoring. , 1996, Archives of surgery.

[124]  E. Peli,et al.  Image invariance with changes in size: the role of peripheral contrast thresholds. , 1991, Journal of the Optical Society of America. A, Optics and image science.

[125]  A. Fernie,et al.  Helmet‐mounted display with dual resolution , 1995 .

[126]  G. Blair Browder.  Evaluation Of A Helmet-Mounted Laser Projector Display , 1989, Defense, Security, and Sensing.

[127]  C. Wickens.  Engineering psychology and human performance, 2nd ed. , 1992 .