Enhancing Ultrasound Haptics with Parametric Audio Effects

Ultrasound haptic devices can create parametric audio as well as contactless haptic feedback. We investigate if multimodal output from these devices can influence the perception of haptic feedback. We used a magnitude estimation experiment to evaluate perceived roughness of an ultrasound haptic pattern. We found that white noise audio from the haptics device increased perceived roughness and pure tones did not, and that lower rendering frequencies may increase perceived roughness. Our results show that multimodal output has the potential to expand the range of sensations that can be presented by an ultrasound haptic device, paving the way to richer mid-air haptic interfaces.

[1]  Shuichi Sakamoto,et al.  Selective effects of auditory stimuli on tactile roughness perception , 2008, Brain Research.

[2]  Hiroyuki Shinoda,et al.  Tactile Stimulation by Repetitive Lateral Movement of Midair Ultrasound Focus , 2020, IEEE Transactions on Haptics.

[3]  Hiroyuki Shinoda,et al.  Non-contact Method for Producing Tactile Sensation Using Airborne Ultrasound , 2008, EuroHaptics.

[4]  Sriram Subramanian,et al.  Perception of ultrasonic haptic feedback on the hand: localisation and apparent motion , 2014, CHI.

[5]  Sriram Subramanian,et al.  Emotions Mediated Through Mid-Air Haptics , 2015, CHI.

[6]  C. Brown.  1965 , 2017, Lie on your wounds.

[7]  Carlos Velasco,et al.  Integrating Mid-Air Haptics into Movie Experiences , 2017, TVX.

[8]  M. Yoneyama,et al.  The audio spotlight: An application of nonlinear interaction of sound waves to a new type of loudspeaker design , 1983 .

[9]  Özgür Izmirli,et al.  Haptic and sound correlations: pitch, loudness and texture , 2004, Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004..

[10]  Sriram Subramanian,et al.  Using Spatiotemporal Modulation to Draw Tactile Patterns in Mid-Air , 2018, EuroHaptics.

[11]  Richard Hayden,et al.  Design and Evaluation of Mid-Air Haptic Interactions in an Augmented Reality Environment , 2018, EuroHaptics.

[12]  Euan Freeman,et al.  Multimodal feedback in HCI: haptics, non-speech audio, and their applications , 2017, The Handbook of Multimodal-Multisensor Interfaces, Volume 1.

[13]  Kasper Hornbæk,et al.  Generating Haptic Textures with a Vibrotactile Actuator , 2017, CHI.

[14]  Claudio Pacchierotti,et al.  Investigating the Recognition of Local Shapes Using Mid-air Ultrasound Haptics , 2019, 2019 IEEE World Haptics Conference (WHC).

[15]  J. F. Dammann,et al.  Temporal Frequency Channels Are Linked across Audition and Touch , 2009, Current Biology.

[16]  Jiro Gyoba,et al.  Effects of sounds on tactile roughness depend on the congruency between modalities , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[17]  H. O. Berktay,et al.  Parametric amplification by the use of acoustic non-linearities and some possible applications , 1965 .

[18]  Orestis Georgiou,et al.  Using Ultrasonic Mid-air Haptic Patterns in Multi-Modal User Experiences , 2019, 2019 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE).

[19]  R. Montaño-Murillo,et al.  GS-PAT: High-Speed Multi-Point Sound-Fields for Phased Arrays of Transducers , 2020 .

[20]  Yoichi Ochiai,et al.  Holographic Whisper: Rendering Audible Sound Spots in Three-dimensional Space by Focusing Ultrasonic Waves , 2017, CHI.

[21]  Orestis Georgiou,et al.  User engagement for mid-air haptic interactions with digital signage , 2019, PerDis.

[22]  J. Kaiser,et al.  Multisensory Object Perception in the Primate Brain , 2010 .

[23]  F. Joseph Pompei,et al.  Sound from ultrasound : the parametric array as an audible sound source , 2002 .

[24]  Sriram Subramanian,et al.  Rendering volumetric haptic shapes in mid-air using ultrasound , 2014, ACM Trans. Graph..

[25]  Orestis Georgiou,et al.  Incorporating the Perception of Visual Roughness into the Design of Mid-Air Haptic Textures , 2020, SAP.

[26]  Orestis Georgiou,et al.  UltraPower: Powering Tangible & Wearable Devices with Focused Ultrasound , 2021, TEI.

[27]  Sriram Subramanian,et al.  Talking about tactile experiences , 2013, CHI.

[28]  Orestis Georgiou,et al.  Mid-Air Haptic Textures from Graphics , 2022 .

[29]  C. Spence,et al.  Audiotactile interactions in roughness perception , 2002, Experimental Brain Research.

[30]  Diego Martinez Plasencia,et al.  A volumetric display for visual, tactile and audio presentation using acoustic trapping , 2019, Nature.

[31]  Stephen Brewster,et al.  Feeling Rough: Multimodal Perception of Virtual Roughness , 2001 .

[32]  Heather Culbertson,et al.  Ungrounded Haptic Augmented Reality System for Displaying Roughness and Friction , 2017, IEEE/ASME Transactions on Mechatronics.

[33]  Lynette A. Jones,et al.  Application of Psychophysical Techniques to Haptic Research , 2013, IEEE Transactions on Haptics.

[34]  Sriram Subramanian,et al.  Sampling Strategy for Ultrasonic Mid-Air Haptics , 2019, CHI.

[35]  Orestis Georgiou,et al.  Mid-Air Haptic Rendering of 2D Geometric Shapes With a Dynamic Tactile Pointer , 2020, IEEE Transactions on Haptics.

[36]  Orestis Georgiou,et al.  Touchless Haptic Feedback for Supernatural VR Experiences , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[37]  Chi Thanh Vi,et al.  Agency in Mid-air Interfaces , 2017, CHI.

[38]  Hiroyuki Shinoda,et al.  Lateral Modulation of Midair Ultrasound Focus for Intensified Vibrotactile Stimuli , 2018, EuroHaptics.

[39]  Euan Freeman,et al.  HaptiGlow: Helping Users Position their Hands for Better Mid-Air Gestures and Ultrasound Haptic Feedback , 2019, 2019 IEEE World Haptics Conference (WHC).

[40]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[41]  Euan Freeman,et al.  Perception of Ultrasound Haptic Focal Point Motion , 2021, ICMI.

[42]  Stephen Brewster,et al.  A Survey of Mid-Air Ultrasound Haptics and Its Applications , 2020, IEEE Transactions on Haptics.

[43]  Yasutoshi Makino,et al.  HaptoClone (Haptic-Optical Clone) for Mutual Tele-Environment by Real-time 3D Image Transfer with Midair Force Feedback , 2016, CHI.

[44]  Seung-Chan Kim,et al.  The Effect of Sound on Haptic Perception , 2007, Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07).

[45]  Stephen A. Brewster,et al.  Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions , 2014, ICMI.

[46]  Susan J. Lederman,et al.  Multisensory Texture Perception , 2010 .

[47]  Mohamad Eid,et al.  Haptogram: Ultrasonic Point-Cloud Tactile Stimulation , 2016, IEEE Access.

[48]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[49]  Stephen A. Brewster,et al.  Textured surfaces for ultrasound haptic displays , 2017, ICMI.

[50]  Sonja Rümelin,et al.  Clicks are in the Air: How to Support the Interaction with Floating Objects through Ultrasonic Feedback , 2017, AutomotiveUI.