The whys, how tos, and pitfalls of user studies

Members of the SIGGRAPH community are both consumers and producers of algorithms that make images and techniques that let us interact with visual applications. This course explains the essential role of user studies in insuring that algorithms, products, and content are effective for their intended purposes. The course introduces user studies through real examples and case studies that highlight good practices and warn of mistakes that can compromise evaluation. The cases are chosen to demonstrate the range of the application of user studies in computer graphics (for examle, in developing better algorithms and in evaluating images and interaction techniques. The first part of the course summarizes the different types of studies that are appropriate at different times during development of a user-interface technique. The second part uses examples from the evaluation of graphics to illustrate the different techniques that can be used to assess perceived image quality and preference. It includes some well-known techniques such as ranking and introduces novel concepts such as maximum-likelihood difference scaling. The third part reviews how eye tracking can support user studies. Eye tracking is useful as an interaction device in virtual environments, but it can also be a helpful tool in usability testing and evaluation of algorithms and techniques. It can collect and analyze additional data that can not be measured explicitly using questionnaires.

[1]  Mel Slater,et al.  Taking steps: the influence of a walking technique on presence in virtual reality , 1995, TCHI.

[2]  Mary C. Whitton,et al.  Passive haptics significantly enhances virtual environments , 2001 .

[3]  Poika Isokoski,et al.  Eye Tracker Input in First Person Shooter Games , 2006 .

[4]  Kenneth Knoblauch,et al.  MLDS: Maximum Likelihood Difference Scaling in R , 2008 .

[5]  Susan K. Schnipke,et al.  Trials and tribulations of using an eye-tracking system , 2000, CHI Extended Abstracts.

[6]  Alan Chalmers,et al.  Detail to Attention: Exploiting Visual Tasks for Selective Rendering , 2003, Rendering Techniques.

[7]  Allen Allport,et al.  Visual attention , 1989 .

[8]  Cornell Juliano,et al.  The hunt for usability: tracking eye movements , 1999, CHI Extended Abstracts.

[9]  Anatole Lécuyer,et al.  Using an Eye-Tracking System to Improve Depth-of-Field Blur Effects and Camera Motions in Virtual Environments , 2008 .

[10]  Mary C. Whitton,et al.  Managing Collaboration in the nanoManipulator , 2003, Presence: Teleoperators & Virtual Environments.

[11]  Diego Gutierrez,et al.  Perceptual rendering of participating media , 2007, TAP.

[12]  T. D. Wilson,et al.  Review of: Berg, Bruce L. Qualitative research methods for the social sciences, 6th ed. Boston, MA: Allyn and Bacon, 2007 , 2008, Inf. Res..

[13]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[14]  M. Whitton,et al.  Review of Four Studies on the Use of Physiological Reaction as a Measure of Presence in StressfulVirtual Environments , 2005, Applied psychophysiology and biofeedback.

[15]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[16]  Robert J. K. Jacob,et al.  What you look at is what you get: eye movement-based interaction techniques , 1990, CHI '90.

[17]  Marilyn Tremaine.  CHI '00 Extended Abstracts on Human Factors in Computing Systems , 2000, CHI Extended Abstracts.

[18]  Mary C. Whitton,et al.  Evaluating a scientific collaboratory: Results of a controlled experiment , 2003, TCHI.

[19]  Mary C. Whitton,et al.  Designing to Support Collaborative Scientific Research Across Distances: The nanoManipulator Environment , 2001, Collaborative Virtual Environments.

[20]  Robert J. K. Jacob,et al.  Eye tracking in human-computer interaction and usability research : Ready to deliver the promises , 2002 .

[21]  Sharif Razzaque,et al.  Comparing VE locomotion interfaces , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[22]  M. Just,et al.  Eye fixations and cognitive processes , 1976, Cognitive Psychology.

[23]  Jeremy M Wolfe,et al.  Visual Attention , 2020, Computational Models for Cognitive Vision.

[24]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[25]  Erik Reinhard,et al.  A psychophysical study of fixation behavior in a computer game , 2008, APGV '08.

[26]  Michael Meehan,et al.  Physiological measures of presence in stressful virtual environments , 2002, SIGGRAPH.

[27]  Mary C. Whitton,et al.  Using innovation diffusion theory to guide collaboration technology evaluation: work in progress , 2001, Proceedings Tenth IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. WET ICE 2001.

[28]  David Connah,et al.  Seeing Beyond Luminance: A Psychophysical Comparison of Techniques for Converting Colour Images to Greyscale , 2007, Color Imaging Conference.

[29]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[30]  D. Spalding.  The Principles of Psychology , 1873, Nature.

[31]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[32]  Mary C. Whitton,et al.  LLCM-WIP: Low-Latency, Continuous-Motion Walking-in-Place , 2008, 2008 IEEE Symposium on 3D User Interfaces.

[33]  Francisco J. Serón,et al.  Non-linear volume photon mapping , 2005, EGSR '05.

[34]  Deborah Hix,et al.  A field study of the Wheel - a usability engineering process model , 2006, J. Syst. Softw..

[35]  Graham J Hole,et al.  How to Design and Report Experiments , 2002 .

[36]  Christophe Charrier,et al.  Maximum likelihood difference scaling of image quality in compression-degraded images. , 2007, Journal of the Optical Society of America. A, Optics, image science, and vision.

[37]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[38]  Heiner Deubel,et al.  The mind's eye : cognitive and applied aspects of eye movement research , 2003 .

[39]  Tutis Vilis.  The Physiology of the Senses Transformations For Perception and Action Lecture 5-The Cerebral Association Cortex , .

[40]  Carol O'Sullivan,et al.  Collisions and Attention , 2005, TAP.

[41]  Veronica Sundstedt,et al.  Gaze and voice based game interaction: the revenge of the killer penguins , 2008, SIGGRAPH '08.

[42]  Samuel Kaski,et al.  Inferring Relevance from Eye Movements: Feature Extraction , 2005 .

[43]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[44]  T. C. Nicholas Graham,et al.  Use of eye movements for video game control , 2006, ACE '06.

[45]  Sungkil Lee,et al.  Real-time tracking of visually attended objects in interactive virtual environments , 2007, VRST '07.

[46]  L. Stark,et al.  Scanpaths in saccadic eye movements while viewing and recognizing patterns. , 1971, Vision research.

[47]  Linden J. Ball,et al.  In Search of Salience: A Response-time and Eye-movement Analysis of Bookmark Recognition , 2004, BCS HCI.

[48]  Gerd Marmitt,et al.  Modeling Visual Attention in VR: Measuring the Accuracy of Predicted Scanpaths , 2002, Eurographics.

[49]  Peter Thompson,et al.  Basic Vision: An Introduction to Visual Perception , 2006 .

[50]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[51]  Mary C. Whitton,et al.  Designing to support situation awareness across distances: an example from a scientific collaboratory , 2004, Inf. Process. Manag..

[52]  Jeff B. Pelz,et al.  Head movement estimation for wearable eye tracker , 2004, ETRA.

[53]  Carol O'Sullivan,et al.  Eye-catching crowds: saliency based selective variation , 2009, ACM Trans. Graph..

[54]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[55]  Karin Veronica Sundstedt.  Rendering and validation of high-fidelity graphics using region-of-interest , 2007 .

[56]  Kristian Lukander,et al.  Mobile usability - Measuring gaze point on handheld devices , 2003 .

[57]  Laurence T Maloney,et al.  Maximum likelihood difference scaling. , 2003, Journal of vision.

[58]  Mary C. Whitton,et al.  Walking > walking-in-place > flying, in virtual environments , 1999, SIGGRAPH.

[59]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[60]  Deborah Hix,et al.  User-centered design and evaluation of a real-time battlefield visualization virtual environment , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[61]  Donald P. Greenberg,et al.  Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments , 2005, TOGS.

[62]  Peter G. Engeldrum,et al.  Psychometric Scaling: A Toolkit for Imaging Systems Development , 2000 .

[63]  Linden J. Ball,et al.  Eye Tracking in Human-Computer Interaction and Usability Research : Current Status and Future Prospects , 2004 .

[64]  Veronica Sundstedt,et al.  Visual attention for efficient high-fidelity graphics , 2005, SCCG '05.

[65]  Sharif Razzaque,et al.  MACBETH: Management of Avatar Conflict by Employment of a Technique Hybrid , 2007, Int. J. Virtual Real..

[66]  Carol O'Sullivan,et al.  Predicting and Evaluating Saliency for Simplified Polygonal Models , 2005, TAP.

[67]  Deborah Hix,et al.  User-Centered Design and Evaluation of Virtual Environments , 1999, IEEE Computer Graphics and Applications.