A Survey of Mid-Air Ultrasound Haptics and Its Applications

Ultrasound haptics is a contactless haptic technology that enables novel mid-air interactions with rich multisensory feedback. This paper surveys recent advances in ultrasound haptic technology. We discuss the fundamentals of this haptic technology, how a variety of perceptible sensations are rendered, and how it is currently being used to enable novel interaction techniques. We summarize its strengths, weaknesses, and potential applications across various domains. We conclude with our perspective on key directions for this promising haptic technology.

[1]  Claudio Pacchierotti,et al.  Investigating the Recognition of Local Shapes Using Mid-air Ultrasound Haptics , 2019, 2019 IEEE World Haptics Conference (WHC).

[2]  Y. Visell,et al.  Emerging Material Technologies for Haptics , 2019, Advanced Materials Technologies.

[3]  C. Davis.  Touch , 1997, The Lancet.

[4]  Mohamad Eid,et al.  Haptogram: Ultrasonic Point-Cloud Tactile Stimulation , 2016, IEEE Access.

[5]  Yasutoshi Makino,et al.  Mid-Air Ultrasonic Pressure Control on Skin by Adaptive Focusing , 2016, EuroHaptics.

[6]  Hannah Limerick,et al.  Call to interact: communicating interactivity and affordances for contactless gesture controlled public displays , 2020, PerDis.

[7]  Mohamad Eid,et al.  Mid-Air Tactile Feedback Co-Located With Virtual Touchscreen Improves Dual-Task Performance , 2020, IEEE Transactions on Haptics.

[8]  Mehdi Ammi,et al.  Non-intrusive Haptic Interfaces: State-of-the Art Survey , 2013, HAID.

[9]  Yasutoshi Makino,et al.  Reducing Amplitude Fluctuation by Gradual Phase Shift in Midair Ultrasound Haptics , 2020, IEEE Transactions on Haptics.

[10]  Lorna M. Brown,et al.  Tactons: Structured Tactile Messages for Non-Visual Information Display , 2004, AUIC.

[11]  David Geerts,et al.  Invisible Touch: How Identifiable are Mid-Air Haptic Shapes? , 2019, CHI Extended Abstracts.

[12]  M. Hanson,et al.  Health effects of exposure to ultrasound and infrasound: report of the independent advisory group on non-ionising radiation , 2010 .

[13]  Abdulmotaleb El-Saddik,et al.  Contactless haptic feedback: state of the art , 2015, 2015 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE).

[14]  Hiroyuki Shinoda,et al.  Thin and flexible airborne ultrasound phased array for tactile display , 2017, 2017 56th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE).

[15]  Mitsuru Ito,et al.  Hybrid Focus Using 70 and 40 kHz Ultrasound in Mid-Air Tactile Display , 2016, AsiaHaptics.

[16]  Ahsan Raza,et al.  Perceptually Correct Haptic Rendering in Mid-Air Using Ultrasound Phased Array , 2020, IEEE Transactions on Industrial Electronics.

[17]  Seungmoon Choi,et al.  Vibrotactile Display: Perception, Technology, and Applications , 2013, Proceedings of the IEEE.

[18]  Yasutoshi Makino,et al.  Midair Haptic Display to Human Upper Body , 2018, 2018 57th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE).

[19]  Euan Freeman,et al.  Multimodal feedback in HCI: haptics, non-speech audio, and their applications , 2017, The Handbook of Multimodal-Multisensor Interfaces, Volume 1.

[20]  Ismo Rakkolainen,et al.  Head-mounted display with mid-air tactile feedback , 2015, VRST.

[21]  Hiroyuki Shinoda,et al.  Aerial Vibrotactile Display Based on Multiunit Ultrasound Phased Array , 2018, IEEE Transactions on Haptics.

[22]  Mitsuru Ito,et al.  High Spatial Resolution Midair Tactile Display Using 70 kHz Ultrasound , 2016, EuroHaptics.

[23]  Hiroyuki Shinoda,et al.  Adding tactile reaction to hologram , 2009, RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication.

[24]  Yasutoshi Makino,et al.  Computer-Created Interactive 3D Image with Midair Haptic Feedback , 2016, AsiaHaptics.

[25]  Orestis Georgiou,et al.  Touchless Haptic Feedback for Supernatural VR Experiences , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[26]  Yasutoshi Makino,et al.  HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback , 2016, CHI.

[27]  Chi Thanh Vi,et al.  Agency in Mid-air Interfaces , 2017, CHI.

[28]  Orestis Georgiou,et al.  User engagement for mid-air haptic interactions with digital signage , 2019, PerDis.

[29]  Sriram Subramanian,et al.  Adding haptic feedback to mobile tv , 2011, CHI Extended Abstracts.

[30]  Arthur P. Berkhoff,et al.  Development of a flexible large-area array based on printed polymer transducers for mid-air haptic feedback , 2019 .

[31]  Mohamad A. Eid,et al.  Characterization of Ultrasound Tactile Display , 2016, EuroHaptics.

[32]  Uma Jayaram,et al.  Weight Sensation in Virtual Environments Using a Haptic Device With Air Jets , 2003, J. Comput. Inf. Sci. Eng..

[33]  Satoshi Saga,et al.  HeatHapt Thermal Radiation-Based Haptic Display , 2014, AsiaHaptics.

[34]  Nigel W. John,et al.  Using and Validating Airborne Ultrasound as a Tactile Interface within Medical Training Simulators , 2014, ISBMS.

[35]  Sriram Subramanian,et al.  Metamaterial bricks and quantization of meta-surfaces , 2016, Nature Communications.

[36]  Weizhi Nai,et al.  Tactile sensitivity in ultrasonic haptics: Do different parts of hand and different rendering methods have an impact on perceptual threshold? , 2019, Virtual Real. Intell. Hardw..

[37]  Hiroyuki Shinoda,et al.  Non-contact tactile sensation synthesized by ultrasound transducers , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[38]  Seungmoon Choi,et al.  Mid-air tactile display using indirect laser radiation for contour-following stimulation and assessment of its spatial acuity , 2017, 2017 IEEE World Haptics Conference (WHC).

[39]  Desney S. Tan,et al.  AirWave: non-contact haptic feedback using air vortex rings , 2013, UbiComp.

[40]  Jan O. Borchers,et al.  FingerFlux: near-surface haptic feedback on tabletops , 2011, UIST.

[41]  D. Ginty,et al.  The Sensory Neurons of Touch , 2013, Neuron.

[42]  Felix Raymond,et al.  Touch hologram in mid-air , 2017, SIGGRAPH Emerging Technologies.

[43]  Yasutoshi Makino,et al.  HORN: the hapt-optic reconstruction , 2014, SIGGRAPH '14.

[44]  T. Hoshi,et al.  UltraPulse - simulating a human arterial pulse with focussed airborne ultrasound , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[45]  Hiroyuki Shinoda,et al.  Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli , 2018, EuroHaptics.

[46]  Orestis Georgiou,et al.  Feel the noise: Mid-air ultrasound haptics as a novel human-vehicle interaction paradigm. , 2019, Applied ergonomics.

[47]  Benjamin Long,et al.  Fibonacci Spiral Arranged Ultrasound Phased Array for Mid-Air Haptics , 2018, 2018 IEEE International Ultrasonics Symposium (IUS).

[48]  Sriram Subramanian,et al.  Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion , 2014, CHI.

[49]  Weiliang Xu,et al.  Multimodal Haptic Display for Virtual Reality: A Survey , 2020, IEEE Transactions on Industrial Electronics.

[50]  Stephen A. Brewster,et al.  Touching the invisible: Localizing ultrasonic haptic cues , 2015, 2015 IEEE World Haptics Conference (WHC).

[51]  Satoshi Hasegawa,et al.  Cross-Field Aerial Haptics: Rendering Haptic Feedback in Air with Light and Acoustic Fields , 2016, CHI.

[52]  Yasutoshi Makino,et al.  Midair Haptic Pursuit , 2019, IEEE Transactions on Haptics.

[53]  Stephen A. Brewster,et al.  May the Force Be with You: Ultrasound Haptic Feedback for Mid-Air Gesture Interaction in Cars , 2018, AutomotiveUI.

[54]  Elia Gatti,et al.  I'm Sensing in the Rain: Spatial Incongruity in Visual-Tactile Mid-Air Stimulation Can Elicit Ownership in VR Users , 2019, CHI.

[55]  Allison M. Okamura,et al.  Haptics: The Present and Future of Artificial Touch Sensation , 2018, Annu. Rev. Control. Robotics Auton. Syst..

[56]  Carlos Velasco,et al.  Integrating Mid-Air Haptics into Movie Experiences , 2017, TVX.

[57]  Roope Raisamo,et al.  A Survey of Mid-Air Ultrasonic Tactile Feedback , 2019, 2019 IEEE International Symposium on Multimedia (ISM).

[58]  L. R. Gavrilov,et al.  Focused ultrasound as a tool to input sensory information to humans (Review) , 2012 .

[59]  Daniel Brice,et al.  A Proof of Concept Integrated Multi-systems Approach for Large Scale Tactile Feedback in VR , 2019, AVR.

[60]  Yasutoshi Makino,et al.  Direct Finger Manipulation of 3D Object Image with Ultrasound Haptic Feedback , 2019, CHI.

[61]  Sriram Subramanian,et al.  Sampling Strategy for Ultrasonic Mid-Air Haptics , 2019, CHI.

[62]  Orestis Georgiou,et al.  Touchless Haptic Feedback for VR Rhythm Games , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[63]  Takayuki Hoshi.  Development of aerial-input and aerial-tactile-feedback system , 2011, 2011 IEEE World Haptics Conference.

[64]  Orestis Georgiou,et al.  Laser Doppler Vibrometry and FEM Simulations of Ultrasonic Mid-Air Haptics , 2019, 2019 IEEE World Haptics Conference (WHC).

[65]  Ismo Rakkolainen,et al.  Bidirectional touch interaction for immaterial displays , 2014, MindTrek.

[66]  Orestis Georgiou,et al.  Cutaneous Wave Propagation Shapes Tactile Motion: Evidence from Air-Coupled Ultrasound , 2019, 2019 IEEE World Haptics Conference (WHC).

[67]  Euan Freeman,et al.  HaptiGlow: Helping Users Position their Hands for Better Mid-Air Gestures and Ultrasound Haptic Feedback , 2019, 2019 IEEE World Haptics Conference (WHC).

[68]  Hiroyuki Shinoda,et al.  Noncontact Tactile Display Based on Radiation Pressure of Airborne Ultrasound , 2010, IEEE Transactions on Haptics.

[69]  Xavier Rottenberg,et al.  Display Compatible PMUT Array for Mid-Air Haptic Feedback , 2019, 2019 20th International Conference on Solid-State Sensors, Actuators and Microsystems & Eurosensors XXXIII (TRANSDUCERS & EUROSENSORS XXXIII).

[70]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[71]  David R. Large,et al.  Exploring the Use of Mid-Air Ultrasonic Feedback to Enhance Automotive User Interfaces , 2018, AutomotiveUI.

[72]  Orestis Georgiou,et al.  Mid-Air Haptic Rendering of 2D Geometric Shapes With a Dynamic Tactile Pointer , 2020, IEEE Transactions on Haptics.

[73]  Ian Oakley,et al.  Whiskers: Exploring the Use of Ultrasonic Haptic Cues on the Face , 2018, CHI.

[74]  Miha Ciglar.  An Ultrasound Based Instrument Generating Audible and Tactile Sound , 2010, NIME.

[75]  Yoichi Ochiai,et al.  Sonovortex: rendering multi-resolution aerial haptics by aerodynamic vortex and focused ultrasound , 2017, SIGGRAPH Posters.

[76]  Hiroyuki Shinoda,et al.  Midair Click of Dual-Layer Haptic Button , 2019, 2019 IEEE World Haptics Conference (WHC).

[77]  Orestis Georgiou,et al.  Mid-Air Haptic Interfaces for Interactive Digital Signage and Kiosks , 2019, CHI Extended Abstracts.

[78]  Jörg Müller,et al.  HaptiRead: Reading Braille as Mid-Air Haptic Information , 2020, Conference on Designing Interactive Systems.

[79]  Chi Thanh Vi,et al.  Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition , 2017, Int. J. Hum. Comput. Stud..

[80]  Anthony J. Croxford,et al.  Realization of Compact Tractor Beams using Acoustic Delay-Lines , 2017 .

[81]  Pan Hui,et al.  A Survey on Haptic Technologies for Mobile Augmented Reality , 2017, ACM Comput. Surv..

[82]  Orestis Georgiou,et al.  Mid-Air Haptic Textures from Graphics , 2022 .

[83]  Małgorzata Pawlaczyk-Łuszczyńska,et al.  Effects of Ultrasonic Noise on the Human Body—A Bibliographic Review , 2013, International journal of occupational safety and ergonomics : JOSE.

[84]  Kenton O'Hara,et al.  Touchless interaction in surgery , 2014, CACM.

[85]  Sriram Subramanian,et al.  Rendering volumetric haptic shapes in mid-air using ultrasound , 2014, ACM Trans. Graph..

[86]  Mohamad A. Eid,et al.  Characterizing tactile rendering parameters for ultrasound based stimulation , 2017, 2017 IEEE World Haptics Conference (WHC).

[87]  Yoichi Ochiai,et al.  Holographic Whisper: Rendering Audible Sound Spots in Three-dimensional Space by Focusing Ultrasonic Waves , 2017, CHI.

[88]  Hiroyuki Shinoda,et al.  Visio-Acoustic screen for contactless touch interface with tactile sensation , 2013, 2013 World Haptics Conference (WHC).

[89]  Hiroyuki Shinoda,et al.  HaptoMime: mid-air haptic interaction with a floating virtual screen , 2014, UIST.

[90]  Hiroyuki Shinoda,et al.  Tactile Stimulation by Repetitive Lateral Movement of Midair Ultrasound Focus , 2020, IEEE Transactions on Haptics.

[91]  Yuriko Suzuki,et al.  Air jet driven force feedback in virtual reality , 2005, IEEE Computer Graphics and Applications.

[92]  Colin H. Hansen,et al.  A Review of Current Ultrasound Exposure Limits , 2004 .

[93]  Bruce W. Drinkwater,et al.  Ultraino: An Open Phased-Array System for Narrowband Airborne Ultrasound Transmission , 2018, IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control.

[94]  Weiliang Xu,et al.  Haptic display for virtual reality: progress and challenges , 2019, Virtual Real. Intell. Hardw..

[95]  Stephen A. Brewster,et al.  Do That, There: An Interaction Technique for Addressing In-Air Gesture Systems , 2016, CHI.

[96]  L. Gavrilov.  The possibility of generating focal regions of complex configurations in application to the problems of stimulation of human receptor structures by focused ultrasound , 2008 .

[97]  Stephen A. Brewster,et al.  Evaluating Ultrasonic Tactile Feedback Stimuli , 2020, EuroHaptics.

[98]  Sriram Subramanian,et al.  Sparkle: Hover Feedback with Touchable Electric Arcs , 2017, CHI.

[99]  Slim Abdennadher,et al.  Rendering of Virtual Volumetric Shapes Using an Electromagnetic-Based Haptic Interface , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[100]  Orestis Georgiou,et al.  Haptic In-Vehicle Gesture Controls , 2017, AutomotiveUI.

[101]  Kentaro Yoshida,et al.  HaptoCloneAR: mutual haptic-optic interactive system with 2D image superimpose , 2017, SIGGRAPH Emerging Technologies.

[102]  Marko LIEBLER,et al.  Quantitative characterization of high-intensity focused airborne ultrasonic fields , 2019 .

[103]  Jong-Rak Park,et al.  Laser-induced thermoelastic effects can evoke tactile sensations , 2015, Scientific Reports.

[104]  Sriram Subramanian,et al.  Using Spatiotemporal Modulation to Draw Tactile Patterns in Mid-Air , 2018, EuroHaptics.

[105]  Claudio Pacchierotti,et al.  PUMAH: Pan-Tilt Ultrasound Mid-Air Haptics for Larger Interaction Workspace in Virtual Reality , 2019, IEEE Transactions on Haptics.

[106]  Richard Hayden,et al.  Design and Evaluation of Mid-Air Haptic Interactions in an Augmented Reality Environment , 2018, EuroHaptics.

[107]  Hiroyuki Shinoda,et al.  Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound , 2008, EuroHaptics.

[108]  Sriram Subramanian,et al.  Emotions Mediated Through Mid-Air Haptics , 2015, CHI.

[109]  Yasutoshi Makino,et al.  Active touch perception produced by airborne ultrasonic haptic hologram , 2015, 2015 IEEE World Haptics Conference (WHC).

[110]  Kim Willems,et al.  Touching the void: Exploring consumer perspectives on touch-enabling technologies in online retailing , 2017 .

[111]  Inwook Hwang,et al.  AirPiano: Enhancing music playing experience in virtual reality with mid-air haptic feedback , 2017, 2017 IEEE World Haptics Conference (WHC).

[112]  Orestis Georgiou,et al.  Mid-Air Haptic Bio-Holograms in Mixed Reality , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[113]  Stephen A. Brewster,et al.  Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions , 2014, ICMI.

[114]  Orestis Georgiou,et al.  Touchless Tactile Displays for Digital Signage: Mid-air Haptics meets Large Screens , 2018, CHI Extended Abstracts.

[115]  Claudio Pacchierotti,et al.  Can Stiffness Sensations Be Rendered in Virtual Reality Using Mid-air Ultrasound Haptic Technologies? , 2020, EuroHaptics.

[116]  Frederick P. Brooks,et al.  Project GROPEHaptic displays for scientific visualization , 1990, SIGGRAPH.

[117]  Hiroyuki Shinoda,et al.  Aerial display of vibrotactile sensation with high spatial-temporal resolution using large-aperture airborne ultrasound phased array , 2013, 2013 World Haptics Conference (WHC).

[118]  Suranga Nanayakkara,et al.  M-Hair: Creating Novel Tactile Feedback by Augmenting the Body Hair to Respond to Magnetic Field , 2019, UIST.

[119]  Stephen A. Brewster,et al.  Textured surfaces for ultrasound haptic displays , 2017, ICMI.

[120]  Christopher J. Plack,et al.  Effects of High-Intensity Airborne Ultrasound Exposure on Behavioural and Electrophysiological Measures of Auditory Function , 2019, Acta Acustica united with Acustica.

[121]  Sonja Rümelin,et al.  Clicks are in the Air: How to Support the Interaction with Floating Objects through Ultrasonic Feedback , 2017, AutomotiveUI.

[122]  Chen Zhao,et al.  Demonstration of Refinity: An Interactive Holographic Signage for New Retail Shopping Experience , 2019, CHI Extended Abstracts.

[123]  Yasutoshi Makino,et al.  Midair Hand Guidance by an Ultrasound Virtual Handrail , 2019, 2019 IEEE World Haptics Conference (WHC).

[124]  Ismo Rakkolainen,et al.  Midair User Interfaces Employing Particle Screens , 2015, IEEE Computer Graphics and Applications.

[125]  Martin L. Lenhardt,et al.  Airborne ultrasonic standards for hearing protection, 2008 , 2008 .

[126]  Gavin Doherty,et al.  Touchless computer interfaces in hospitals: A review , 2018, Health Informatics J..

[127]  Jong-Rak Park,et al.  Mid-air tactile stimulation using laser-induced thermoelastic effects: The first study for indirect radiation , 2015, 2015 IEEE World Haptics Conference (WHC).

[128]  Haiwen Luan,et al.  Skin-integrated wireless haptic interfaces for virtual and augmented reality , 2019, Nature.

[129]  Shun Suzuki,et al.  Noncontact Thermal and Vibrotactile Display Using Focused Airborne Ultrasound , 2020, EuroHaptics.

[130]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.