Multiwave: Complex Hand Gesture Recognition Using the Doppler Effect

We built an acoustic, gesture-based recognition system called Multiwave, which leverages the Doppler Effect to translate multidimensional movements into user interface commands. Our system only requires the use of a speaker and microphone to be operational, but can be augmented with more speakers. Since these components are already included in most end user systems, our design makes gesture-based input more accessible to a wider range of end users. We are able to detect complex gestures by generating a known high frequency tone from multiple speakers and detecting movement using changes in the sound waves. We present the results of a user study of Multiwave to evaluate recognition rates for different gestures and report error rates comparable to or better than the current state of the art despite additional complexity. We also report subjective user feedback and some lessons learned from our system that provide additional insight for future applications of multidimensional acoustic gesture recognition.

[1]  Shyamnath Gollakota,et al.  Enabling on-body transmissions with commodity devices , 2016, UbiComp.

[2]  E. Owens,et al.  An Introduction to the Psychology of Hearing , 1997 .

[3]  Shwetak N. Patel,et al.  AirLink: sharing files between multiple devices using in-air gestures , 2014, UbiComp.

[4]  Yael Edan,et al.  Vision-based hand-gesture applications , 2011, Commun. ACM.

[5]  Joseph J. LaViola,et al.  3D Gesture classification with linear acceleration and angular velocity sensing devices for video games , 2013, Entertain. Comput..

[6]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[7]  Joseph J. LaViola,et al.  Penny pincher: a blazing fast, highly accurate $-family recognizer , 2015, Graphics Interface.

[8]  Pamela J. Wisniewski,et al.  Multiwave: Doppler Effect Based Gesture Recognition in Multiple Dimensions , 2016, CHI Extended Abstracts.

[9]  Desney S. Tan,et al.  Your noise is my command: sensing gestures using the body as an antenna , 2011, CHI.

[10]  Yang Xu,et al.  WiFinger: talk to your smart devices with finger-grained gesture , 2016, UbiComp.

[11]  Eric C. Larson,et al.  DopLink: using the doppler effect for multi-device interaction , 2013, UbiComp.

[12]  Bhiksha Raj,et al.  One-handed gesture recognition using ultrasonic Doppler sonar , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[13]  Mo Li,et al.  DopEnc: acoustic-based encounter profiling using smartphones , 2016, MobiCom.

[14]  Shwetak N. Patel,et al.  SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal , 2014, UIST.

[15]  Lawrence E. Kinsler,et al.  Fundamentals of acoustics , 1950 .

[16]  Majed Al Zayer,et al.  PAWdio: Hand Input for Mobile VR using Acoustic Sensing , 2016, CHI PLAY.

[17]  Joseph J. LaViola,et al.  An introduction to 3D gestural interfaces , 2014, SIGGRAPH '14.

[18]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[19]  Desney S. Tan,et al.  SoundWave: using the doppler effect to sense gestures , 2012, CHI.

[20]  Lisa Anthony,et al.  A lightweight multistroke recognizer for user interface prototypes , 2010, Graphics Interface.

[21]  Greg Welch,et al.  Motion Tracking: No Silver Bullet, but a Respectable Arsenal , 2002, IEEE Computer Graphics and Applications.

[22]  Tsutomu Terada,et al.  Ultrasound-based movement sensing, gesture-, and context-recognition , 2013, ISWC '13.

[23]  Joseph J. LaViola,et al.  Jackknife: A Reliable Recognizer with Few Samples and Many Modalities , 2017, CHI.

[24]  Radu Ionescu,et al.  3D Localization and Tracking of Objects Using Miniature Microphones , 2011, Wirel. Sens. Netw..

[25]  Lei Yang,et al.  AudioGest: enabling fine-grained hand gesture detection by decoding echo signal , 2016, UbiComp.