Discovering Users’ Intent in Interactive Virtual Environments

A virtual reality(VR) environment is defined as a computer generated representation of reality that is sensitive to the actions of its observer. As the computing power of our machines follows an ever growing trend, the simulation power of our VR applications and their impact on the development of our society continues to grow in a remarkable fashion. Along with our computing capabilities, the data that needs to be spatially manipulated continuously increases in size and diversity. To keep up with this trend of increasing complexity we need to develop new 3D user interfaces (3DUIs) that allow users to employ the full manipulative capabilities of their natural hand gestures when manipulating such data. Today we can approach this goal by tracking the natural hand gestures of our users and inferring their manipulative intentions. However, human natural hand gestures exhibit a large variability that is aggravated by hand placement inaccuracies and body tracking uncertainties. Additionally, there is a non-unique mapping between human gestures and the underlying manipulative intentions. In this dissertation I lay out the foundation of a general manipulative intention inference framework. New metrics are proposed for quantifying a set of human behavioral cues that characterize general goal directed actions. The relationship between these behavioral cues and a user’s manipulative intent is modeled using machine learning techniques in novel fashion. The practical value of these techniques is demonstrated by developing new virtual object manipulation methods that are driven by intention inference. By means of intention inference, the proposed interaction techniques automatically adapt to the user’s subjective needs for various enhancements such as hand placement fault tolerance and hand positioning precision enhancement. The performance of the resulting virtual object manipulation techniques has been tested in a statistically significant Frol Periverzov University of Connecticut, 2016 manner by means of user studies. The work presented here advances the state of the art in 3DUIs towards more user-friendly or even person centered user interfaces by developing user adaptable interfaces driven by intention inference. This can dramatically shorten the time required by a novice user to start performing efficient virtual object manipulations. Discovering Users’ Intent in Interactive Virtual

[1]  Q. M. Jonathan Wu,et al.  3D Shape from Focus and Depth Map Computation Using Steerable Filters , 2009, ICIAR.

[2]  Jinsong Bao,et al.  Assembly operation process planning by mapping a virtual assembly simulation to real operation , 2013, Comput. Ind..

[3]  Beryl Plimmer,et al.  3D input for 3D worlds , 2007, OZCHI '07.

[4]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[5]  Christopher Zach,et al.  High-Performance Multi-View Reconstruction , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[6]  L. Tian,et al.  Quantitative measurement of size and three-dimensional position of fast-moving bubbles in air-water mixture flows using digital holography. , 2010, Applied optics.

[7]  Douglas Lanman,et al.  Build your own 3D scanner: 3D photography for beginners , 2009, SIGGRAPH '09.

[8]  F. Althoff,et al.  ROBUST MULTIMODAL HAND-AND HEAD GESTURE RECOGNITION FOR CONTROLLING AUTOMOTIVE INFOTAINMENT SYSTEMS , 2005 .

[9]  Anna PelagottiMelania Digital holography for 3D imaging and display in the IR range: challenges and opportunities , 2010 .

[10]  Xiumin Fan,et al.  Virtual assembly with physical information: a review , 2015 .

[11]  Stephan Wong,et al.  Vision­Based Hand Gesture Recognition for Human Computer Interaction: A Review , 2008 .

[12]  G. Lippmann Epreuves reversibles donnant la sensation du relief , 1908 .

[13]  Hema Swetha Koppula,et al.  Learning human activities and object affordances from RGB-D videos , 2012, Int. J. Robotics Res..

[14]  Judy M. Vance,et al.  Virtual reality for assembly methods prototyping: a review , 2011, Virtual Reality.

[15]  Song Zhang,et al.  High-resolution, real-time 3D imaging with fringe analysis , 2010, Journal of Real-Time Image Processing.

[16]  Yuuki Watanabe,et al.  Profilometry with compact single-shot low-coherence time-domain interferometry , 2008 .

[17]  Jr. Joseph J. LaViola,et al.  A Survey of Hand Posture and Gesture Recognition Techniques and Technology , 1999 .

[18]  Bernd Fröhlich,et al.  A generalized God-object method for plausible finger-based interactions in virtual environments , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[19]  Helge Ritter,et al.  Motor synergies and object representations in virtual and real grasping , 2010 .

[20]  Angeline M. Loh,et al.  Shape from Non-homogeneous, Non-stationary, Anisotropic, Perspective Texture , 2005, BMVC.

[21]  Pascal Picart,et al.  Real-time three-sensitivity measurements based on three-color digital Fresnel holographic interferometry. , 2010, Optics letters.

[22]  Giovanna Sansoni,et al.  State-of-The-Art and Applications of 3D Imaging Sensors in Industry, Cultural Heritage, Medicine, and Criminal Investigation , 2009, Sensors.

[23]  Mircea Nicolescu,et al.  Vision-based hand pose estimation: A review , 2007, Comput. Vis. Image Underst..

[24]  Kevin George Harding,et al.  3D imaging using a unique refractive optic design to combine moire and stereo , 1997, Other Conferences.

[25]  Ashok Veeraraghavan,et al.  Structured light 3D scanning in the presence of global illumination , 2011, CVPR 2011.

[26]  Andreas Nüchter,et al.  3D Robotic Mapping - The Simultaneous Localization and Mapping Problem with Six Degrees of Freedom , 2009, Springer Tracts in Advanced Robotics.

[27]  Thorsten Joachims,et al.  Cutting-plane training of structural SVMs , 2009, Machine Learning.

[28]  Klaus Bengler,et al.  A Usability Study on Hand Gesture Controlled Operation of In-Car Devices , 2001 .

[29]  Olivier D. Faugeras,et al.  Shape From Shading , 2006, Handbook of Mathematical Models in Computer Vision.

[30]  Christoph W. Borst,et al.  A Spring Model for Whole-Hand Virtual Grasping , 2006, Presence: Teleoperators & Virtual Environments.

[31]  Ferran Argelaguet,et al.  A survey of 3D object selection techniques for virtual environments , 2013, Comput. Graph..

[32]  Andrea Thelen,et al.  Improvements in Shape-From-Focus for Holographic Reconstructions With Regard to Focus Operators, Neighborhood-Size, and Height Value Interpolation , 2009, IEEE Transactions on Image Processing.

[33]  Scott Frees,et al.  Context-driven interaction in immersive virtual environments , 2010, Virtual Reality.

[34]  Dominique Bechmann,et al.  Starfish: a selection technique for dense virtual environments , 2012, VRST '12.

[35]  Ivan Poupyrev,et al.  The go-go interaction technique: non-linear mapping for direct manipulation in VR , 1996, UIST '96.

[36]  Alison Gopnik,et al.  Toddlers' understanding of intentions, desires and emotions: Explorations of the dark ages. , 1999 .

[37]  S. Glover,et al.  Separate visual representations in the planning and control of action , 2004, Behavioral and Brain Sciences.

[38]  Joong-Sun Won,et al.  Mapping Three-Dimensional Surface Deformation by Combining Multiple-Aperture Interferometry and Conventional Interferometry: Application to the June 2007 Eruption of Kilauea Volcano, Hawaii , 2011, IEEE Geoscience and Remote Sensing Letters.

[39]  Joseph J. LaViola,et al.  Dense and Dynamic 3D Selection for Game-Based Virtual Environments , 2012, IEEE Transactions on Visualization and Computer Graphics.

[40]  A. N. Rajagopalan,et al.  Dealing With Parallax in Shape-From-Focus , 2011, IEEE Transactions on Image Processing.

[41]  Anthony Steed,et al.  Towards a General Model for Selection in Virtual Environments , 2006, 3D User Interfaces (3DUI'06).

[42]  Thomas Feix,et al.  A comprehensive grasp taxonomy , 2009 .

[43]  Robert Lange,et al.  3D time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology , 2006 .

[44]  Stephan Hussmann,et al.  A Performance Review of 3D TOF Vision Systems in Comparison to Stereo Vision Systems , 2008 .

[45]  T. Latychevskaia,et al.  Solution to the twin image problem in holography. , 2006, Physical review letters.

[46]  Christian C. Wagner,et al.  Survey on classifying human actions through visual sensors , 2012, Artificial Intelligence Review.

[47]  Bernd Fröhlich,et al.  Effective manipulation of virtual objects within arm's reach , 2011, 2011 IEEE Virtual Reality Conference.

[48]  Rafael Radkowski,et al.  Interactive Hand Gesture-based Assembly for Augmented Reality Applications , 2012, ACHI 2012.

[49]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[50]  Grigore C. Burdea,et al.  Virtual reality technology (2. ed.) , 2003 .

[51]  Torsten Kuhlen,et al.  Multi-Contact Grasp Interaction for Virtual Environments , 2008, J. Virtual Real. Broadcast..

[52]  Joan Lasenby,et al.  Shape from Texture Via Fourier Analysis , 2008, ISVC.

[53]  Doug A. Bowman,et al.  Rapid and accurate 3D selection by progressive refinement , 2011, 2011 IEEE Symposium on 3D User Interfaces (3DUI).

[54]  R. S. Jadon,et al.  A REVIEW OF VISION BASED HAND GESTURES RECOGNITION , 2009 .

[55]  Soh-Khim Ong,et al.  Augmented reality aided interactive manual assembly design , 2013, The International Journal of Advanced Manufacturing Technology.

[56]  Archana P. Sangole,et al.  Palmar arch dynamics during reach-to-grasp tasks , 2008, Experimental Brain Research.

[57]  Carlos Hernández,et al.  Practical 3D Reconstruction Based on Photometric Stereo , 2010, Computer Vision: Detection, Recognition and Reconstruction.

[58]  J. Todd,et al.  The perception of 3D shape from texture based on directional width gradients. , 2010, Journal of vision.

[59]  C. Heyes,et al.  Infants' behavioral reenactment of "failed attempts": exploring the roles of emulation learning, stimulus enhancement, and understanding of intentions. , 2002, Developmental psychology.

[60]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Marko Koch,et al.  Combining 3D Laser-Scanning and Close-Range Photogrammetry - An Approach to Exploit the Strength of Both Methods , 2009 .

[62]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[63]  Young-June Kang,et al.  A study on the 3-D measurement by using digital projection moiré method , 2008 .

[64]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[65]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[66]  Trevor Darrell,et al.  Latent-Dynamic Discriminative Models for Continuous Gesture Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Mahendra P. Kothiyal,et al.  Two-wavelength micro-interferometry for 3-D surface profiling , 2009 .

[68]  S. Foix,et al.  Lock-in Time-of-Flight (ToF) Cameras: A Survey , 2011, IEEE Sensors Journal.

[69]  Dimitrios E. Maroulis,et al.  A real-time FPGA architecture for 3D reconstruction from integral images , 2010, J. Vis. Commun. Image Represent..

[70]  Adrian Bradu,et al.  Extra long imaging range swept source optical coherence tomography using re-circulation loops. , 2010, Optics express.

[71]  D. F. Howell,et al.  Coordinate measurement in 2-D and 3-D geometries using frequency scanning interferometry , 2005 .

[72]  Frol Periverzov,et al.  IDS: The intent driven selection method for natural user interfaces , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[73]  Rémi Ronfard,et al.  A survey of vision-based methods for action representation, segmentation and recognition , 2011, Comput. Vis. Image Underst..

[74]  Terrence Fernando,et al.  A constraint manager to support virtual maintainability , 2003, Comput. Graph..

[75]  Roberto A. Braga,et al.  Sensitivity of the moiré technique for measuring biological surfaces , 2008 .

[76]  R. Danescu,et al.  High accuracy stereo vision system for far distance obstacle detection , 2004, IEEE Intelligent Vehicles Symposium, 2004.

[77]  Sebastian Thrun,et al.  3D shape scanning with a time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[78]  J Arai,et al.  Real-time pickup method for a three-dimensional image based on integral photography. , 1997, Applied optics.

[79]  Tamar Flash,et al.  Motor primitives in vertebrates and invertebrates , 2005, Current Opinion in Neurobiology.

[80]  Thierry Oggier,et al.  CCD / CMOS Lock-In Pixel for Range Imaging : Challenges , Limitations and State-ofthe-Art , 2005 .

[81]  Joaquim Salvi,et al.  A state of the art in structured light patterns for surface profilometry , 2010, Pattern Recognit..

[82]  Liang-Chia Chen,et al.  3-D surface profilometry using simultaneous phase-shifting interferometry , 2010 .

[83]  Bahram Javidi,et al.  Performance of 3D integral imaging with position uncertainty. , 2007, Optics express.

[84]  Fatemeh Mohammadi,et al.  Application of Digital Phase Shift Moiré to Reconstruction of Human Face , 2010, 2010 Fourth UKSim European Symposium on Computer Modeling and Simulation.

[85]  Stefano Soatto,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE IEEE TRANSACTION OF PATTERN RECO , 2022 .

[86]  David A. Forsyth,et al.  Shape from Texture without Boundaries , 2002, International Journal of Computer Vision.

[87]  Joseph Rosen,et al.  Review of three-dimensional holographic imaging by Fresnel incoherent correlation holograms , 2010 .

[88]  Ying Wu,et al.  Vision-Based Gesture Recognition: A Review , 1999, Gesture Workshop.

[89]  Krassimir Dotchev,et al.  Dimensional error analysis in point cloud-based inspection using a non-contact method for data acquisition , 2010 .

[90]  Yaser Sheikh,et al.  Shape from Dynamic Texture for Planes , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[91]  G. Drew Kessler,et al.  PRISM interaction for enhancing control in immersive virtual environments , 2007, TCHI.

[92]  Craig S. Chapman,et al.  To use or to move: goal-set modulates priming when grasping real tools , 2011, Experimental Brain Research.

[93]  Yasushi Yagi,et al.  Dynamic scene shape reconstruction using a single structured light pattern , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[94]  Anind K. Dey,et al.  Navigate like a cabbie: probabilistic reasoning from observed context-aware behavior , 2008, UbiComp.

[95]  Marco Santello,et al.  Patterns of Hand Motion during Grasping and the Influence of Sensory Guidance , 2002, The Journal of Neuroscience.

[96]  G. Yahav,et al.  3D Imaging Camera for Gaming Application , 2007, 2007 Digest of Technical Papers International Conference on Consumer Electronics.

[97]  Imre Horváth,et al.  HAND MOTION PROCESSING IN APPLICATIONS: A CONCISE SURVEY AND ANALYSIS OF TECHNOLOGIES , 2004 .

[98]  Reinhard Koch,et al.  Time‐of‐Flight Cameras in Computer Graphics , 2010, Comput. Graph. Forum.

[99]  Noritaka Osawa,et al.  Adjustment and control methods for precise rotation and positioning of virtual object by hand , 2010, VRCAI '10.

[100]  S. Kita,et al.  The nature of gestures ' beneficial role in spatial problem solving , 2013 .

[101]  A. Podoleanu,et al.  Optical coherence tomography , 2012, Journal of microscopy.

[102]  Sam Van der Jeught,et al.  Implementation of phase-shifting moiré profilometry on a low-cost commercial data projector , 2010 .

[103]  Yael Edan,et al.  Designing Hand Gesture Vocabularies for Natural Interaction by Combining Psycho-Physiological and Recognition Factors , 2008, Int. J. Semantic Comput..

[104]  Mindy F Levin,et al.  A New Perspective in the Understanding of Hand Dysfunction Following Neurological Injury , 2007, Topics in stroke rehabilitation.

[105]  Bernd Fröhlich,et al.  Natural Interaction Metaphors for Functional Validations of Virtual Car Models , 2011, IEEE Transactions on Visualization and Computer Graphics.

[106]  Gal A. Kaminka,et al.  Towards computational models of intention detection and intention prediction , 2014, Cognitive Systems Research.

[107]  G. Csibra,et al.  Teleological reasoning in infancy: the naı̈ve theory of rational action , 2003, Trends in Cognitive Sciences.

[108]  Karin Coninx,et al.  Exploring the Effects of Environment Density and Target Visibility on Object Selection in 3D Virtual Environments , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[109]  Sanjeev Sofat,et al.  Vision Based Hand Gesture Recognition , 2009 .

[110]  François Blais Review of 20 years of range sensor development , 2004, J. Electronic Imaging.

[111]  Qican Zhang,et al.  Dynamic 3-D shape measurement method: A review , 2010 .

[112]  Henry A. Kautz,et al.  Learning and inferring transportation routines , 2004, Artif. Intell..

[113]  Andrew Blake,et al.  Efficient Dense Stereo with Occlusions for New View-Synthesis by Four-State Dynamic Programming , 2006, International Journal of Computer Vision.

[114]  M. B. Ahmad,et al.  Focus measure operator using 3D gradient , 2007, 2007 International Conference on Machine Vision.

[115]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[116]  Gang Ren,et al.  3D selection with freehand gesture , 2013, Comput. Graph..

[117]  Heni Ben Amor,et al.  Grasp Recognition with Uncalibrated Data Gloves - A Comparison of Classification Methods , 2007, 2007 IEEE Virtual Reality Conference.

[118]  Martin Schaffer,et al.  High-speed three-dimensional shape measurements of objects with laser speckles and acousto-optical deflection. , 2011, Optics letters.

[119]  Christian Laugier,et al.  Intentional motion on-line learning and prediction , 2008, Machine Vision and Applications.

[120]  M. Tomasello,et al.  Fourteen-through 18-month-old infants di eren-tially imitate intentional and accidental actions , 1998 .

[121]  M. Santello,et al.  The role of vision on hand preshaping during reach to grasp , 2003, Experimental Brain Research.

[122]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[123]  Lei Liu,et al.  Insights from Dividing 3D Goal-Directed Movements into Meaningful Phases , 2009, IEEE Computer Graphics and Applications.

[124]  Jason Geng,et al.  Structured-light 3D surface imaging: a tutorial , 2011 .

[125]  Frits H. Post,et al.  IntenSelect: using dynamic object rating for assisting 3D object selection , 2005, EGVE'05.

[126]  Young-Koo Lee,et al.  Semi-Markov conditional random fields for accelerometer-based activity recognition , 2010, Applied Intelligence.

[127]  Tanja Schultz,et al.  Combined intention, activity, and motion recognition for a humanoid household robot , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[128]  Jong-An Park,et al.  Shape from Focus through Laplacian Using 3D Window , 2008, 2008 Second International Conference on Future Generation Communication and Networking.

[129]  Alice Biber,et al.  Time-of-flight range imaging with a custom solid state image sensor , 1999, Industrial Lasers and Inspection.

[130]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[131]  J. Valença,et al.  Applications of photogrammetry to structural assessment , 2012, Experimental Techniques.

[132]  Lik-Kwan Shark,et al.  Immersive manipulation of virtual objects through glove-based hand gesture interaction , 2011, Virtual Reality.

[133]  James J. Filliben,et al.  Characterization of the Range Performance of a 3D Imaging System (NIST TN 1695) | NIST , 2011 .

[134]  Martin Schaffer,et al.  High-speed pattern projection for three-dimensional shape measurement using laser speckles. , 2010, Applied optics.