A Window to Your Smartphone: Exploring Interaction and Communication in Immersive VR with Augmented Virtuality

A major drawback of most Head Mounted Displays (HMDs) used in immersive Virtual Reality (VR) is the visual and social isolation of users from their real-world surroundings while wearing these headsets. This partial isolation of users from the real-world might hinder social interactions with friends and family. To address this issue, we present a new method to allow people wearing VR HMDs to use their smartphones or tablets without removing their HMDs. To do this, we augment the scene inside the VR HMD with a view of the user's device so that the user can interact with the device without removing the headset. The idea involves the use of additional cameras, such as the Leap Motion device or a high-resolution RGB camera to capture the user's real-world surrounding and augment the virtual world with the content displayed on the smartphone screen. This setup allows VR users to have a window to their smartphone from within the virtual world and afford all of the functionality provided by their smartphones, with the potential to reduce some of the undesirable isolation users may experience when using immersive VR systems.

[1]  Maria T. Schultheis,et al.  Head-Mounted Displays for Clinical Virtual Reality Applications: Pitfalls in Understanding User Behavior while Using Technology , 2006, Cyberpsychology Behav. Soc. Netw..

[2]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Guannan Gao,et al.  Probabilistic Hough Transform , 2011 .

[4]  Roderick Murray-Smith,et al.  A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays , 2015, CHI Extended Abstracts.

[5]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[6]  Paul Milgram,et al.  A Taxonomy of Real and Virtual World Display Integration , 1999 .

[7]  D. Böhning.  Multinomial logistic regression algorithm , 1992 .

[8]  Banshidhar Majhi,et al.  Classification of brain MR images using discrete wavelet transform and random forests , 2015, 2015 Fifth National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics (NCVPRIPG).

[9]  Clark F. Olson,et al.  Automatic target recognition by matching oriented edge pixels , 1997, IEEE Trans. Image Process..

[10]  Klaus H. Hinrichs,et al.  Immersive virtual studio for architectural exploration , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[11]  Hirokazu Kato,et al.  Collaborative Mixed Reality , 1999 .

[12]  Rodrigo Garrido.  HTC One (M8) , 2014 .

[13]  Aman S. Mathur,et al.  Low cost virtual reality for medical training , 2015, 2015 IEEE Virtual Reality (VR).

[14]  David Kushner.  Virtual reality's moment , 2014, IEEE Spectrum.

[15]  Tom Drummond,et al.  Dynamic measurement clustering to aid real time tracking , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[16]  Jeffrey S. Norris,et al.  Immersive and collaborative data visualization using virtual reality platforms , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[17]  James P. Bliss,et al.  The Effectiveness of Virtual Reality for Administering Spatial Navigation Training to Firefighters , 1997, Presence: Teleoperators & Virtual Environments.

[18]  James A. G. Henry,et al.  The effects of immersion and navigation on the acquisition of spatial knowledge of abstract data networks , 2010, ICCS.

[19]  Henrik I. Christensen,et al.  3D textureless object detection and tracking: An edge-based approach , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Erik Champion,et al.  The Palenque project: evaluating interaction in an online virtual archaeology site , 2011, Virtual Reality.

[21]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[22]  Trevor Hastie,et al.  An Introduction to Statistical Learning , 2013, Springer Texts in Statistics.

[23]  Vincent Lepetit,et al.  Gradient Response Maps for Real-Time Detection of Textureless Objects , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Federico Tombari,et al.  BOLD Features to Detect Texture-less Objects , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Holger Regenbrecht,et al.  Using Augmented Virtuality for Remote Collaboration , 2004, Presence: Teleoperators & Virtual Environments.

[26]  Massimo Bergamasco,et al.  I'm in VR!: using your own hands in a fully immersive MR system , 2014, VRST '14.

[27]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[28]  Daniela M. Witten,et al.  An Introduction to Statistical Learning: with Applications in R , 2013 .

[29]  Klaus H. Hinrichs,et al.  Enhancing Presence in Head-Mounted Display Environments by Visual Body Feedback Using Head-Mounted Cameras , 2009, 2009 International Conference on CyberWorlds.

[30]  Yoav Freund,et al.  A Short Introduction to Boosting , 1999 .

[31]  Le Hoang Thai,et al.  Facial Expression Classification Using Artificial Neural Network and K-Nearest Neighbor , 2015 .

[32]  Ao Li,et al.  A novel method for user-defined human posture recognition using Kinect , 2014, 2014 7th International Congress on Image and Signal Processing.

[33]  Kemao Qian,et al.  BORDER: An Oriented Rectangles Approach to Texture-Less Object Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Umberto Castellani,et al.  Machine Learning Approaches: From Theory to Application in Schizophrenia , 2013, Comput. Math. Methods Medicine.

[36]  David Nahon,et al.  “Never Blind VR” enhancing the virtual reality headset experience with augmented virtuality , 2015, 2015 IEEE Virtual Reality (VR).

[37]  C. Steger.  OCCLUSION , CLUTTER , AND ILLUMINATION INVARIANT OBJECT RECOGNITION , 2002 .

[38]  P. Danielsson.  Euclidean distance mapping , 1980 .

[39]  Klaus H. Hinrichs,et al.  Poster: A virtual body for augmented virtuality by chroma-keying of egocentric videos , 2009, 2009 IEEE Symposium on 3D User Interfaces.

[40]  覃政.  Google Cardboard：伟大的搅局者 , 2014 .

[41]  Vincent Lepetit,et al.  Combining edge and texture information for real-time accurate 3D camera tracking , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[42]  Koldo Basterretxea,et al.  Efficient Algorithms for Accelerometer-Based Wearable Hand Gesture Recognition Systems , 2015, 2015 IEEE 13th International Conference on Embedded and Ubiquitous Computing.

[43]  Thomas Hofmann,et al.  Support Vector Machines for Multiple-Instance Learning , 2002, NIPS.

[44]  Kenneth J Ottenbacher,et al.  A virtual reality environment for evaluation of a daily living skill in brain injury rehabilitation: reliability and validity. , 2003, Archives of physical medicine and rehabilitation.

[45]  Victoria Interrante,et al.  Towards Achieving Robust Video Self-avatars under Flexible Environment Conditions , 2014 .

[46]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[47]  Max Kuhn,et al.  Building Predictive Models in R Using the caret Package , 2008 .

[48]  Sukhan Lee,et al.  Robust Recognition and Pose Estimation of 3D Objects Based on Evidence Fusion in a Sequence of Images , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[49]  Sudeb Das,et al.  Brain Mr Image Classification Using Multiscale Geometric Analysis of Ripplet , 2013 .

[50]  Rainer Groh,et al.  Aughanded Virtuality - the hands in the virtual environment , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[51]  Robert C. Bolles,et al.  Parametric Correspondence and Chamfer Matching: Two New Techniques for Image Matching , 1977, IJCAI.

[52]  Helena Rua,et al.  Living the past: 3D models, virtual reality and game engines as tools for supporting archaeology and the reconstruction of cultural heritage – the case-study of the Roman villa of Casal de Freiria , 2011 .

[53]  Takeo Kanade,et al.  A statistical method for 3D object detection applied to faces and cars , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[54]  Robert W. Lindeman.  A low-cost, low-latency approach to dynamic immersion in occlusive head-mounted displays , 2016, 2016 IEEE Virtual Reality (VR).

[55]  Maya R. Gupta,et al.  Bayesian Quadratic Discriminant Analysis , 2007, J. Mach. Learn. Res..

[56]  Markus Ulrich,et al.  Combining Scale-Space and Similarity-Based Aspect Graphs for Fast 3D Object Recognition , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Daniel Buschek,et al.  Automatic Privacy Classification of Personal Photos , 2015, INTERACT.

[58]  Daniel T. Larose,et al.  k‐Nearest Neighbor Algorithm , 2005 .

[59]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[60]  B. Scholkopf,et al.  Fisher discriminant analysis with kernels , 1999, Neural Networks for Signal Processing IX: Proceedings of the 1999 IEEE Signal Processing Society Workshop (Cat. No.98TH8468).

[61]  Steven K. Feiner,et al.  WeARHand: Head-worn, RGB-D camera-based, bare-hand user interface with visually enhanced depth perception , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[62]  Paul J. Metzger.  Adding reality to the virtual , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[63]  Akihiro Takeuchi,et al.  On-road vehicle tracking using deformable object model and particle filter with integrated likelihoods , 2010, 2010 IEEE Intelligent Vehicles Symposium.

[64]  David G. Lowe,et al.  Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration , 2009, VISAPP.

[65]  Vincent Lepetit,et al.  Dominant orientation templates for real-time detection of texture-less objects , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[66]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[67]  Brendan Iribe.  Oculus Rift를 이용한 체감형 게임 구현 , 2014 .

[68]  Csaba Beleznai,et al.  GPU-Accelerated Human Detection Using Fast Directional Chamfer Matching , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[69]  F. Hausdorff.  Dimension und äußeres Maß , 1918 .

[70]  Ting Wang,et al.  Application of Breiman's Random Forest to Modeling Structure-Activity Relationships of Pharmaceutical Molecules , 2004, Multiple Classifier Systems.

[71]  Hélder P. Oliveira,et al.  A Kinect-Based System to Assess Lymphedema Impairments in Breast Cancer Patients , 2015, IbPRIA.