Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality

Advances in display and tracking technologies hold the promise of increasingly immersive augmented-reality experiences. Unfortunately, the on-demand generation of haptic experiences is lagging behind these advances in other feedback channels. We present Annexing Reality; a system that opportunistically annexes physical objects from a user's current physical environment to provide the best-available haptic sensation for virtual objects. It allows content creators to a priori specify haptic experiences that adapt to the user's current setting. The system continuously scans user's surrounding, selects physical objects that are similar to given virtual objects, and overlays the virtual models on to selected physical ones reducing the visual-haptic mismatch. We describe the developer's experience with the Annexing Reality system and the techniques utilized in realizing it. We also present results of a developer study that validates the usability and utility of our method of defining haptic experiences.

[1]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[2]  S. Weghorst,et al.  Virtual reality and tactile augmentation in the treatment of spider phobia: a case report. , 1997, Behaviour research and therapy.

[3]  Christopher M. Brown,et al.  Haptics in augmented reality , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[4]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Arindam Dey,et al.  BurnAR: Feel the heat , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  Jan O. Borchers,et al.  Instant user interfaces: repurposing everyday objects as input devices , 2013, ITS.

[7]  Peter Kulchyski.  and , 2015 .

[8]  Hans-Werner Gellersen,et al.  Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences , 2015, CHI.

[9]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[10]  Marc Alexa,et al.  Computing and Rendering Point Set Surfaces , 2003, IEEE Trans. Vis. Comput. Graph..

[11]  Thomas H. Massie,et al.  The PHANToM Haptic Interface: A Device for Probing Virtual Objects , 1994 .

[12]  Gábor Székely,et al.  Visuo-haptic collaborative augmented reality ping-pong , 2007, ACE '07.

[13]  Maud Marchal,et al.  The Virtual Mitten: A novel interaction paradigm for visuo-haptic manipulation of objects using grip force , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[14]  K. Höhne,et al.  Volume Cutting for Virtual Petrous Bone Surgery , 2002, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[15]  Yuriko Suzuki,et al.  Air jet driven force feedback in virtual reality , 2005, IEEE Computer Graphics and Applications.

[16]  Sean White,et al.  Developing an augmented reality racing game , 2008, INTETAIN '08.

[17]  Ulrich Eck,et al.  ClonAR: Rapid redesign of real-world objects , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[18]  Takuji Narumi,et al.  Modifying an Identified Angle of Edged Shapes Using Pseudo-haptic Effects , 2012, EuroHaptics.

[19]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[20]  Takuji Narumi,et al.  Modifying an identified curved surface shape using pseudo-haptic effect , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[21]  Hunter G. Hoffman,et al.  Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[22]  Ramesh Raskar,et al.  Feel through window: simultaneous geometry and texture display based on lateral force , 2012, SIGGRAPH 2012.

[23]  Steven K. Feiner,et al.  Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality , 2008, VRST '08.

[24]  Hiroshi Ishii,et al.  Sublimate: state-changing virtual and physical rendering to augment interaction with shape displays , 2013, CHI.

[25]  Benjamin Lok,et al.  Virtual Human + Tangible Interface = Mixed Reality Human An Initial Exploration with a Virtual Breast Exam Patient , 2008, 2008 IEEE Virtual Reality Conference.

[26]  Sy-Yen Kuo,et al.  iCon: utilizing everyday objects as additional, auxiliary and instant tabletop controllers , 2010, CHI.

[27]  Michitaka Hirose,et al.  Providing force feedback in virtual environments , 1995, IEEE Computer Graphics and Applications.

[28]  HoshiTakayuki,et al.  Fairy Lights in Femtoseconds , 2016 .

[29]  L.M. Di Diodato,et al.  A Haptic Force Feedback Device for Virtual Reality-fMRI Experiments , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[30]  M. Sato,et al.  SPIDAR and virtual reality , 2002, Proceedings of the 5th Biannual World Automation Congress.

[31]  Takuji Narumi,et al.  Modifying an identified position of edged shapes using pseudo-haptic effects , 2012, VRST '12.

[32]  Pattie Maes,et al.  Smarter objects: using AR technology to program physical objects and their interactions , 2013, CHI Extended Abstracts.

[33]  Hiroo Iwata,et al.  Project FEELEX: adding haptic surface to graphics , 2001, SIGGRAPH.

[34]  Ken Hinckley,et al.  Passive real-world interface props for neurosurgical visualization , 1994, CHI '94.

[35]  Radu Bogdan Rusu,et al.  Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments , 2010, KI - Künstliche Intelligenz.

[36]  Desney S. Tan,et al.  AirWave: non-contact haptic feedback using air vortex rings , 2013, UbiComp.

[37]  Henrik Gordon Petersen,et al.  Pose estimation using local structure-specific shape and appearance context , 2013, 2013 IEEE International Conference on Robotics and Automation.

[38]  Ali Israr,et al.  AIREAL: interactive tactile experiences in free air , 2013, ACM Trans. Graph..

[39]  Reinhard Klein,et al.  Efficient RANSAC for Point‐Cloud Shape Detection , 2007, Comput. Graph. Forum.

[40]  Eduardo Velloso,et al.  Substitutional reality , 2015, XRDS.

[41]  Gerard Jounghyun Kim,et al.  Effects of sizes and shapes of props in tangible augmented reality , 2009, 2009 8th IEEE International Symposium on Mixed and Augmented Reality.

[42]  Markus Funk,et al.  An augmented workplace for enabling user-defined tangibles , 2014, CHI Extended Abstracts.

[43]  H. Kuhn.  The Hungarian method for the assignment problem , 1955 .

[44]  Andrea Giachetti,et al.  A multiprocessor decoupled system for the simulation of temporal bone surgery , 2002 .