Direct Finger Manipulation of 3D Object Image with Ultrasound Haptic Feedback

In this study, we prototype and examine a system that allows a user to manipulate a 3D virtual object with multiple fingers without wearing any device. An autostereoscopic display produces a 3D image and a depth sensor measures the movement of the fingers. When a user touches a virtual object, haptic feedback is provided by ultrasound phased arrays. By estimating the cross section of the finger in contact with the virtual object and by creating a force pattern around it, it is possible for the user to recognize the position of the surface relative to the finger. To evaluate our system, we conducted two experiments to show that the proposed feedback method is effective in recognizing the object surface and thereby enables the user to grasp the object quickly without seeing it.

[1]  Mohamad Eid,et al.  Haptogram: Ultrasonic Point-Cloud Tactile Stimulation , 2016, IEEE Access.

[2]  Ismo Rakkolainen,et al.  Head-mounted display with mid-air tactile feedback , 2015, VRST.

[3]  Yasutoshi Makino,et al.  HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback , 2016, CHI.

[4]  Mitsuru Ito,et al.  High Spatial Resolution Midair Tactile Display Using 70 kHz Ultrasound , 2016, EuroHaptics.

[5]  J. Flusser,et al.  Numerically Stable Direct Least Squares Fitting of Ellipses , 1998 .

[6]  John Kenneth Salisbury,et al.  A constraint-based god-object method for haptic display , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[7]  Hideki Kakeya,et al.  A High Quality Autostereoscopy System Based on Time-Division Quadplexing Parallax Barrier , 2014, IEICE Trans. Electron..

[8]  Hiroyuki Shinoda,et al.  HaptoMime: mid-air haptic interaction with a floating virtual screen , 2014, UIST.

[9]  L. Gavrilov.  The possibility of generating focal regions of complex configurations in application to the problems of stimulation of human receptor structures by focused ultrasound , 2008 .

[10]  David Kim,et al.  HoloDesk: direct 3d interactions with a situated see-through display , 2012, CHI.

[11]  Seungju Han,et al.  Holo-Haptics: Haptic interaction with a see-through 3D display , 2014, 2014 IEEE International Conference on Consumer Electronics (ICCE).

[12]  Ali Israr,et al.  AIREAL: interactive tactile experiences in free air , 2013, ACM Trans. Graph..

[13]  C. J. Keemink,et al.  Envelope detection of amplitude-modulated high-frequency sinusoidal signals by skin mechanoreceptors. , 1986, The Journal of the Acoustical Society of America.

[14]  Hideki Kakeya,et al.  [Papers] Time-Division Quadruplexing Parallax Barrier with Subpixel-Based Slit Control , 2018 .

[15]  W. Harwin,et al.  Improved Haptic Rendering for Multi-Finger Manipulation Using Friction Cone based God-Objects , 2002 .

[16]  Hiroyuki Shinoda,et al.  A method for distribution control of aerial ultrasound radiation pressure for remote vibrotactile display , 2013, The SICE Annual Conference 2013.

[17]  Yasutoshi Makino,et al.  Active touch perception produced by airborne ultrasonic haptic hologram , 2015, 2015 IEEE World Haptics Conference (WHC).

[18]  Hiroyuki Shinoda,et al.  Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound , 2008, EuroHaptics.

[19]  Yasutoshi Makino,et al.  Scalable Architecture for Airborne Ultrasound Tactile Display , 2016, AsiaHaptics.

[20]  R. Halír.  Numerically Stable Direct Least Squares Fitting of Ellipses , 1998 .

[21]  Sriram Subramanian,et al.  Rendering volumetric haptic shapes in mid-air using ultrasound , 2014, ACM Trans. Graph..

[22]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[23]  Frédo Durand,et al.  A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach , 2006, International Journal of Computer Vision.

[24]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[25]  Hiroyuki Shinoda,et al.  Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli , 2018, EuroHaptics.

[26]  David Kim,et al.  Vermeer: direct interaction with a 360° viewable 3D display , 2011, UIST.

[27]  Yasutoshi Makino,et al.  Mid-Air Ultrasonic Pressure Control on Skin by Adaptive Focusing , 2016, EuroHaptics.

[28]  Gábor Székely,et al.  High Precision Augmented Reality Haptics , 2006 .

[29]  Yi-Pai Huang,et al.  Bare Finger 3D Air-Touch System Using an Embedded Optical Sensor Array for Mobile Displays , 2014, Journal of Display Technology.

[30]  Maud Marchal,et al.  The god-finger method for improving 3D interaction with virtual objects through simulation of contact area , 2013, 2013 IEEE Symposium on 3D User Interfaces (3DUI).

[31]  Sho Kamuro,et al.  RePro3D: full-parallax 3D display with haptic feedback using retro-reflective projection technology , 2011, 2011 IEEE International Symposium on VR Innovation.

[32]  Yi-Pai Huang,et al.  3D Air-Touch User Interface With High Touch Accuracy on Stereoscopic Displays , 2016, Journal of Display Technology.

[33]  Eun-Soo Kim,et al.  Vision-Based 3D Fingertip Interface for Spatial Interaction in 3D Integral Imaging System , 2009, 2009 International Conference on Complex, Intelligent and Software Intensive Systems.