A survey of 3D object selection techniques for virtual environments

Computer graphics applications controlled through natural gestures are gaining increasing popularity these days due to recent developments in low-cost tracking systems and gesture recognition technologies. Although interaction techniques through natural gestures have already demonstrated their benefits in manipulation, navigation and avatar-control tasks, effective selection with pointing gestures remains an open problem. In this paper we survey the state-of-the-art in 3D object selection techniques. We review important findings in human control models, analyze major factors influencing selection performance, and classify existing techniques according to a number of criteria. Unlike other components of the application's user interface, pointing techniques need a close coupling with the rendering pipeline, introducing new elements to be drawn, and potentially modifying the object layout and the way the scene is rendered. Conversely, selection performance is affected by rendering issues such as visual feedback, depth perception, and occlusion management. We thus review existing literature paying special attention to those aspects in the boundary between computer graphics and human-computer interaction.

[1]  Florence Tyndiuk,et al.  Cognitive comparison of 3D interaction in front of large vs. small displays , 2005, VRST '05.

[2]  Andy Cockburn,et al.  Human on-line response to visual and motor target expansion , 2006, Graphics Interface.

[3]  Frank Vahid,et al.  Enabling nonexpert construction of basic sensor-based systems , 2009, TCHI.

[4]  Klaus H. Hinrichs,et al.  2d touching of 3d stereoscopic objects , 2011, CHI.

[5]  Andrew S. Forsberg,et al.  Aperture based selection for immersive virtual environments , 1996, UIST '96.

[6]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[7]  Ivan Poupyrev,et al.  Manipulating Objects in Virtual Worlds: Categorization and Empirical Evaluation of Interaction Techniques , 1999, J. Vis. Lang. Comput..

[8]  Raimund Dachselt,et al.  A survey and taxonomy of 3D menu techniques , 2006, EGVE'06.

[9]  G. Drew Kessler,et al.  PRISM interaction for enhancing control in immersive virtual environments , 2007, TCHI.

[10]  Harald Reiterer,et al.  Adaptive Pointing - Design and Evaluation of a Precision Enhancing Technique for Absolute Pointing Devices , 2009, INTERACT.

[11]  Shumin Zhai,et al.  The “Silk Cursor”: investigating transparency for 3D target acquisition , 1994, CHI '94.

[12]  Shumin Zhai,et al.  The influence of muscle groups on performance of multiple degree-of-freedom input , 1996, CHI.

[13]  Joseph J. LaViola,et al.  A natural, tiered and executable UIDL for 3D user interfaces based on Concept-Oriented Design , 2009, TCHI.

[14]  Scott R. Klemmer,et al.  Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology , 2011, UIST 2011.

[15]  K CardStuart,et al.  A morphological analysis of the design space of input devices , 1991 .

[16]  Ivan Poupyrev,et al.  A framework and testbed for studying manipulation techniques for immersive VR , 1997, VRST '97.

[17]  Benjamin B. Bederson,et al.  Fisheye menus , 2000, UIST '00.

[18]  Tovi Grossman,et al.  Pointing at trivariate targets in 3D environments , 2004, CHI.

[19]  Karin Coninx,et al.  Using the Non-Dominant Hand for Selection in 3D , 2006, 3D User Interfaces (3DUI'06).

[20]  Daniel Vogel,et al.  Distant freehand pointing and clicking on very large, high resolution displays , 2005, UIST.

[21]  Steven K. Feiner,et al.  SenseShapes: using statistical geometry for object selection in a multimodal augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[22]  Doug A. Bowman,et al.  Exploring individual differences in raybased selection: strategies and traits , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[23]  Niklas Elmqvist,et al.  BalloonProbe: reducing occlusion in 3D using interactive space distortion , 2005, VRST '05.

[24]  Ferran Argelaguet,et al.  Virtual Pads: Decoupling Motor Space and Visual Space for Flexible Manipulation of 2D Windows within VEs , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[25]  Mark R. Mine,et al.  Virtual Environment Interaction Techniques , 1995 .

[26]  Frans Vos,et al.  WYSIWYP: What You See Is What You Pick , 2012, IEEE Transactions on Visualization and Computer Graphics.

[27]  Frederick P. Brooks,et al.  Moving objects in space: exploiting proprioception in virtual-environment interaction , 1997, SIGGRAPH.

[28]  Timo Ropinski,et al.  Object Selection in Virtual Environments using an Improved Virtual Pointer Metaphor , 2004, ICCVG.

[29]  Michael Gleicher,et al.  The challenges of 3D interaction: a CHI '94 workshop , 1994, SGCH.

[30]  Naren Ramakrishnan,et al.  Towards Preferences in Virtual Environment Interfaces , 2002, EGVE.

[31]  H. Zelaznik,et al.  Motor-output variability: a theory for the accuracy of rapid motor acts. , 1979, Psychological review.

[32]  Evangelos A. Christou,et al.  Motor Output Variability , 2010 .

[33]  I. Scott MacKenzie,et al.  Accuracy measures for evaluating computer pointing devices , 2001, CHI.

[34]  Adam Finkelstein,et al.  Adaptive cutaways for comprehensible rendering of polygonal scenes , 2008, SIGGRAPH Asia '08.

[35]  Doug A. Bowman,et al.  The Virtual Venue: User-Computer Interaction in Information-Rich Virtual Environments , 1998, Presence.

[36]  Richard A. Bolt,et al.  “Put-that-there”: Voice and gesture at the graphics interface , 1980, SIGGRAPH '80.

[37]  Donald B. Johnson,et al.  Testbed Evaluation of Virtual Environment Interaction Techniques , 1999, Presence: Teleoperators & Virtual Environments.

[38]  Steven K. Feiner,et al.  The Flexible Pointer: An Interaction Technique for Selection in Augmented and Virtual Reality , 2003 .

[39]  Scott P. Robertson,et al.  Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1991 .

[40]  James F. Cremer,et al.  Distance perception in real and virtual environments , 2004 .

[41]  Doug A. Bowman,et al.  An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments , 1997, SI3D.

[42]  Anthony Steed,et al.  Towards a General Model for Selection in Virtual Environments , 2006, 3D User Interfaces (3DUI'06).

[43]  Gerrit C. van der Veer,et al.  Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems , 1993 .

[44]  Matthias Bues,et al.  iSith - Intersection-based Spatial Interaction for Two Hands , 2006, 3D User Interfaces (3DUI'06).

[45]  C. MacKenzie,et al.  Three-Dimensional Movement Trajectories in Fitts' Task: Implications for Control , 1987 .

[46]  Ferran Argelaguet,et al.  Improving 3D Selection in VEs through Expanding Targets and Forced Disocclusion , 2008, Smart Graphics.

[47]  ConinxKarin,et al.  Adaptation in virtual environments , 2011 .

[48]  R A Abrams,et al.  Optimality in human motor performance: ideal control of rapid aimed movements. , 1988, Psychological review.

[49]  Ravin Balakrishnan,et al.  Reaching for objects in VR displays: lag and frame rate , 1994, TCHI.

[50]  Robert J. K. Jacob,et al.  Interacting with eye movements in virtual environments , 2000, CHI.

[51]  Fernando Argelaguet Sanz,et al.  Pointing facilitation techniques for 3d object selection on virtual environments , 2011 .

[52]  Robert J. Teather,et al.  Effects of tracking technology, latency, and spatial jitter on object movement , 2009, 2009 IEEE Symposium on 3D User Interfaces.

[53]  Ferran Argelaguet,et al.  Overcoming eye-hand visibility mismatch in 3D pointing selection , 2008, VRST '08.

[54]  P. Fitts,et al.  INFORMATION CAPACITY OF DISCRETE MOTOR RESPONSES. , 1964, Journal of experimental psychology.

[55]  Jinseok Seo,et al.  Evaluation of pointing techniques for ray casting selection in virtual environments , 2003, International Conference On Virtual Reality and Its Applications in Industry.

[56]  Andrew S. Forsberg,et al.  Image plane interaction techniques in 3D immersive environments , 1997, SI3D.

[57]  Ferran Argelaguet,et al.  Hand-Based Disocclusion for the World-in-Miniature Metaphor , 2010, PRESENCE: Teleoperators and Virtual Environments.

[58]  WareColin,et al.  Reaching for objects in VR displays , 1994 .

[59]  Anthony Steed,et al.  3D Selection Strategies for Head Tracked and Non-Head Tracked Operation of Spatially Immersive Displays , 2004 .

[60]  Niklas Elmqvist,et al.  View-projection animation for 3D occlusion management , 2007, Comput. Graph..

[61]  Tovi Grossman,et al.  Going Deeper: a Taxonomy of 3D on the Tabletop , 2007, Tabletop.

[62]  Colin Ware,et al.  Selection using a one-eyed cursor in a fish tank VR environment , 1997, TCHI.

[63]  R J Jagacinski,et al.  Fitts' Law in two dimensions with hand and head movements. , 1983, Journal of motor behavior.

[64]  Jock D. Mackinlay,et al.  A morphological analysis of the design space of input devices , 1991, TOIS.

[65]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[66]  Frits H. Post,et al.  IntenSelect: using dynamic object rating for assisting 3D object selection , 2005, EGVE'05.

[67]  Bobby Bodenheimer,et al.  Synthesis and evaluation of linear motion transitions , 2008, TOGS.

[68]  Andriy Pavlovych,et al.  The tradeoff between spatial jitter and latency in pointing tasks , 2009, EICS '09.

[69]  P. Fitts The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[70]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[71]  Nguyen Thong Dang A Survey and Classification of 3D Pointing Techniques , 2007, 2007 IEEE International Conference on Research, Innovation and Vision for the Future.

[72]  Thies Pfeiffer Towards a linguistically motivated model for selection in virtual reality , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[73]  Joseph L. Gabbard A Taxonomy of Usability Characteristics in Virtual Environments , 1997 .

[74]  Randy Pausch,et al.  Virtual reality on a WIM: interactive worlds in miniature , 1995, CHI '95.

[75]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[76]  Joseph J. LaViola,et al.  A discussion of cybersickness in virtual environments , 2000, SGCH.

[77]  Robert W. Lindeman,et al.  Hand-held windows: towards effective 2D interaction in immersive virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[78]  Ivan Poupyrev,et al.  The go-go interaction technique: non-linear mapping for direct manipulation in VR , 1996, UIST '96.

[79]  A BowmanDoug,et al.  A survey of usability evaluation in virtual environments , 2002 .

[80]  Adam Finkelstein,et al.  Adaptive cutaways for comprehensible rendering of polygonal scenes , 2008, SIGGRAPH 2008.

[81]  Shumin Zhai,et al.  Hand eye coordination patterns in target selection , 2000, ETRA.

[82]  Joseph J. LaViola,et al.  Dense and Dynamic 3D Selection for Game-Based Virtual Environments , 2012, IEEE Transactions on Visualization and Computer Graphics.

[83]  Gerard P. van Galen,et al.  Fitts' law as the outcome of a dynamic noise filtering model of motor control , 1995 .

[84]  Laurent Grisoni,et al.  The design and evaluation of 3D positioning techniques for multi-touch displays , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[85]  Ferran Argelaguet,et al.  Efficient 3D Pointing Selection in Cluttered Virtual Environments , 2009, IEEE Computer Graphics and Applications.

[86]  T. Ichikawa,et al.  Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques , 1998, Comput. Graph. Forum.

[87]  Doug A. Bowman,et al.  Rapid and accurate 3D selection by progressive refinement , 2011, 2011 IEEE Symposium on 3D User Interfaces (3DUI).

[88]  Christian Müller-Tomfelde,et al.  Dwell-Based Pointing in Applications of Human Computer Interaction , 2007, INTERACT.

[89]  William Ribarsky,et al.  Evaluation of the effects of frame time variation on VR task performance , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[90]  Alexander Kulik Building on Realism and Magic for Designing 3D Interaction Techniques , 2009, IEEE Computer Graphics and Applications.

[91]  Joseph J. LaViola,et al.  Reflecting on the Design and Implementation Issues of Virtual Environments , 2010, PRESENCE: Teleoperators and Virtual Environments.

[92]  Mark Green,et al.  JDCAD: A highly interactive 3D modeling system , 1994, Comput. Graph..

[93]  Andrew T. Duchowski,et al.  Gaze- vs. hand-based pointing in virtual environments , 2003, CHI Extended Abstracts.

[94]  Ravin Balakrishnan,et al.  "Beating" Fitts' law: virtual enhancements for pointing facilitation , 2004, Int. J. Hum. Comput. Stud..

[95]  Doug A. Bowman,et al.  A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods , 2002, Presence: Teleoperators & Virtual Environments.

[96]  Karin Coninx,et al.  Evaluating the Influence of Multimodal Feedback on Egocentric Selection Metaphors in Virtual Environments , 2006, HAID.

[97]  Ferran Argelaguet,et al.  Visual feedback techniques for virtual pointing on stereoscopic displays , 2009, VRST '09.

[98]  A BowmanDoug,et al.  The Virtual Venue , 1998 .

[99]  Karin Coninx,et al.  Exploring the Effects of Environment Density and Target Visibility on Object Selection in 3D Virtual Environments , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[100]  Ferran Argelaguet,et al.  Anisomorphic ray-casting manipulation for interacting with 2D GUIs , 2007, Comput. Graph..

[101]  Ken Hinckley,et al.  A survey of design issues in spatial input , 1994, UIST '94.

[102]  Doug A. Bowman,et al.  Overcoming World in Miniature Limitations by a Scaled and Scrolling WIM , 2006, 3D User Interfaces (3DUI'06).

[103]  Doug A. Bowman,et al.  Testbed Evaluation of Virtual Environment Interaction Techniques , 2001, Presence Teleoperators Virtual Environ..

[104]  G. Drew Kessler,et al.  Precise and rapid interaction through scaled manipulation in immersive virtual environments , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[105]  Doug A. Bowman,et al.  Evaluating the Need for Display-Specific and Device-Specific 3D Interaction Techniques , 2007, HCI.

[106]  Andy Cockburn,et al.  Multimodal feedback for the acquisition of small targets , 2005, Ergonomics.

[107]  Niklas Elmqvist,et al.  A Taxonomy of 3D Occlusion Management for Visualization , 2008, IEEE Transactions on Visualization and Computer Graphics.

[108]  BalakrishnanRavin "Beating" Fitts' law , 2004 .

[109]  Ferran Argelaguet,et al.  Improving co-located collaboration with show-through techniques , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[110]  Michael Gleicher,et al.  The challenges of 3D interaction , 1994, CHI Conference Companion.

[111]  Doug A. Bowman,et al.  A human motor behavior model for distal pointing tasks , 2010, Int. J. Hum. Comput. Stud..

[112]  A BoltRichard,et al.  Put-that-there , 1980 .

[113]  Robert J. Teather,et al.  Pointing at 3D targets in a stereo head-tracked virtual environment , 2011, 2011 IEEE Symposium on 3D User Interfaces (3DUI).

[114]  I.,et al.  Fitts' Law as a Research and Design Tool in Human-Computer Interaction , 1992, Hum. Comput. Interact..

[115]  Manojit Sarkar,et al.  Graphical fisheye views of graphs , 1992, CHI.

[116]  Daniel Vogel,et al.  The Impact of Control-Display Gain on User Performance in Pointing Tasks , 2008, Hum. Comput. Interact..

[117]  Tovi Grossman,et al.  The design and evaluation of selection techniques for 3D volumetric displays , 2006, UIST.

[118]  Karin Coninx,et al.  Adaptation in virtual environments: conceptual framework and user models , 2010, Multimedia Tools and Applications.

[119]  Atsuo Murata,et al.  Extending Effective Target Width in Fitts' Law to a Two-Dimensional Pointing Task , 1999, Int. J. Hum. Comput. Interact..

[120]  J. Edward Swan,et al.  Toward Disambiguating Multiple Selections for Frustum-Based Pointing , 2006, 3D User Interfaces (3DUI'06).

[121]  Robert W. Lindeman,et al.  Towards usable VR: an empirical study of user interfaces for immersive virtual environments , 1999, CHI '99.

[122]  Yanqing Wang,et al.  Effects of Orientation Disparity Between Haptic and Graphic Displays of Objects in Virtual Environments , 1999, INTERACT.

[123]  J. M. Gerzso,et al.  Computer graphics and interactive techniques: 15th-17th July 1974. Boulder, Colorado, USA. Sponsored by the University of Colorado Computing Centre and ACM/SIGGRAPH , 1975, Comput. Aided Des..

[124]  Ken Hinckley,et al.  Passive real-world interface props for neurosurgical visualization , 1994, CHI '94.

[125]  Doug A. Bowman,et al.  Using Pinch Gloves(TM) for both Natural and Abstract Interaction Techniques in Virtual Environments , 2001 .