A survey on acoustic sensing

The rise of Internet-of-Things (IoT) has brought many new sensing mechanisms. Among these mechanisms, acoustic sensing attracts much attention in recent years. Acoustic sensing exploits acoustic sensors beyond their primary uses, namely recording and playing, to enable interesting applications and new user experience. In this paper, we present the first survey of recent advances in acoustic sensing using commodity hardware. We propose a general framework that categorizes main building blocks of acoustic sensing systems. This framework consists of three layers, i.e., the physical layer, processing layer, and application layer. We highlight different sensing approaches in the processing layer and fundamental design considerations in the physical layer. Many existing and potential applications including context-aware applications, human-computer interface, and aerial acoustic communications are presented in depth. Challenges and future research trends are also discussed.

[1]  Archan Misra,et al.  BreathPrint: Breathing Acoustics-based User Authentication , 2017, MobiSys.

[2]  Jie Yang,et al.  Push the limit of WiFi based localization for smartphones , 2012, Mobicom '12.

[3]  Kang G. Shin,et al.  EchoTag: Accurate Infrastructure-Free Indoor Location Tagging with Smartphones , 2015, MobiCom.

[4]  Lusheng Ji,et al.  Location-Aware IEEE 802.11 for Spatial Reuse Enhancement , 2007, IEEE Transactions on Mobile Computing.

[5]  Romit Roy Choudhury,et al.  BackDoor: Making Microphones Hear Inaudible Sounds , 2017, MobiSys.

[6]  Lei Yang,et al.  Tagoram: real-time tracking of mobile RFID tags to high precision using COTS devices , 2014, MobiCom.

[7]  Silvio Savarese,et al.  Learning to Track: Online Multi-object Tracking by Decision Making , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[8]  Anthony Rowe,et al.  Indoor pseudo-ranging of mobile devices using ultrasonic chirps , 2012, SenSys '12.

[9]  Jie Xiong,et al.  ArrayTrack: A Fine-Grained Indoor Location System , 2011, NSDI.

[10]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Yi-Ping Hung,et al.  RunPlay: Action Recognition Using Wearable Device Apply on Parkour Game , 2016, UIST.

[12]  Qiang Li,et al.  Auditeur: a mobile-cloud service platform for acoustic event detection on smartphones , 2013, MobiSys '13.

[13]  Charles E. Cook,et al.  Linear FM Signal Formats for Beacon and Communication Systems , 1974, IEEE Transactions on Aerospace and Electronic Systems.

[14]  J. D. Lipson,et al.  Chinese remainder and interpolation algorithms , 1971, SYMSAC '71.

[15]  Buntarou Shizuki,et al.  Sensing Touch Force using Active Acoustic Sensing , 2015, TEI.

[16]  Eric C. Larson,et al.  SpiroSmart: using a microphone to measure lung function on a mobile phone , 2012, UbiComp.

[17]  Boaz Rafaely,et al.  Analysis and design of spherical microphone arrays , 2005, IEEE Transactions on Speech and Audio Processing.

[18]  Sunghyun Choi,et al.  Chirp signal-based aerial acoustic communication for smart devices , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[19]  Sanjay Jha,et al.  Received signal strength indicator and its analysis in a typical WLAN system (short paper) , 2013, 38th Annual IEEE Conference on Local Computer Networks.

[20]  David Chu,et al.  SwordFight: enabling a new class of phone-to-phone action games on commodity phones , 2012, MobiSys '12.

[21]  Iain Murray,et al.  Human activity recognition using thigh angle derived from single thigh mounted IMU data , 2014, 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[22]  Swarun Kumar,et al.  Accurate indoor localization with zero start-up cost , 2014, MobiCom.

[23]  Samuel S. Blackman,et al.  Multiple-Target Tracking with Radar Applications , 1986 .

[24]  Kasper Hornbæk,et al.  Expressive touch: studying tapping force on tabletops , 2014, CHI.

[25]  Tinne Tuytelaars,et al.  Modeling video evolution for action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Stefania Sesia,et al.  LTE - The UMTS Long Term Evolution, Second Edition , 2011 .

[27]  Desney S. Tan,et al.  FingerIO: Using Active Sonar for Fine-Grained Finger Tracking , 2016, CHI.

[28]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[29]  Hamed Ketabdar,et al.  MagiTact: interaction with mobile devices based on compass (magnetic) sensor , 2010, IUI '10.

[30]  Kang G. Shin,et al.  Expansion of Human-Phone Interface By Sensing Structure-Borne Sound Propagation , 2016, MobiSys.

[31]  S. Boll,et al.  Suppression of acoustic noise in speech using spectral subtraction , 1979 .

[32]  Venkata N. Padmanabhan,et al.  Centaur: locating devices in an office environment , 2012, Mobicom '12.

[33]  Kung Yao,et al.  Source localization and beamforming , 2002, IEEE Signal Process. Mag..

[34]  Xiaoli Ma,et al.  Timing and Frequency Synchronization for OFDM Downlink Transmissions Using Zadoff-Chu Sequences , 2015, IEEE Transactions on Wireless Communications.

[35]  Tomohiro Nakatani,et al.  Microphone-location dependent mask estimation for BSS using spatially distributed asynchronous microphones , 2013, 2013 International Symposium on Intelligent Signal Processing and Communication Systems.

[36]  Eric Pottier,et al.  A review of target decomposition theorems in radar polarimetry , 1996, IEEE Trans. Geosci. Remote. Sens..

[37]  Tamer Nadeem,et al.  RF-Beep: A light ranging scheme for smart devices , 2013, 2013 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[38]  Ramarathnam Venkatesan,et al.  Dhwani: secure peer-to-peer acoustic NFC , 2013, SIGCOMM.

[39]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[40]  Xiaolin Li,et al.  Guoguo: enabling fine-grained indoor localization via smartphone , 2013, MobiSys '13.

[41]  Desney S. Tan,et al.  SoundWave: using the doppler effect to sense gestures , 2012, CHI.

[42]  Romit Roy Choudhury,et al.  Inaudible Voice Commands: The Long-Range Attack and Defense , 2018, NSDI.

[43]  Chen Wang,et al.  Fine-grained sleep monitoring: Hearing your breathing with smartphones , 2015, 2015 IEEE Conference on Computer Communications (INFOCOM).

[44]  Lei Xie,et al.  VSkin: Sensing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals , 2018, MobiCom.

[45]  Shyamnath Gollakota,et al.  Contactless Sleep Apnea Detection on Smartphones , 2015, GetMobile Mob. Comput. Commun..

[46]  Cordelia Schmid,et al.  P-CNN: Pose-Based CNN Features for Action Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[47]  Wei Wang,et al.  Depth Aware Finger Tapping on Virtual Displays , 2018, MobiSys.

[48]  Wei Wang,et al.  Understanding and Modeling of WiFi Signal Based Human Activity Recognition , 2015, MobiCom.

[49]  Uroschanit Yodprasit,et al.  A 60-GHz SiGe BiCMOS Monostatic Transceiver for FMCW Radar Applications , 2017, IEEE Transactions on Microwave Theory and Techniques.

[50]  S. Chiba,et al.  Dynamic programming algorithm optimization for spoken word recognition , 1978 .

[51]  Dana Ron,et al.  Chinese remaindering with errors , 1999, STOC '99.

[52]  Dimitrios Koutsonikolas,et al.  Messages behind the sound: real-time hidden acoustic signal capture with smartphones , 2016, MobiCom.

[53]  Mi Zhang,et al.  BodyBeat: a mobile system for sensing non-speech body sounds , 2014, MobiSys.

[54]  Bing Zhou,et al.  BatMapper: Acoustic Sensing Based Indoor Floor Plan Construction Using Smartphones , 2017, MobiSys.

[55]  Bruno Sinopoli,et al.  ALPS: A Bluetooth and Ultrasound Platform for Mapping and Localization , 2015, SenSys.

[56]  Yang Xu,et al.  WiFinger: talk to your smart devices with finger-grained gesture , 2016, UbiComp.

[57]  Juha Röning,et al.  MyoGym: introducing an open gym data set for activity recognition collected using myo armband , 2017, UbiComp/ISWC Adjunct.

[58]  Qiang Li,et al.  MusicalHeart: a hearty way of listening to music , 2012, SenSys '12.

[59]  Shoji Makino,et al.  Blind compensation of inter-channel sampling frequency mismatch with maximum likelihood estimation in STFT domain , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[60]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[61]  W. Gregg,et al.  On the Utility of Chirp Modulation for Digital Signaling , 1973, IEEE Trans. Commun..

[62]  Jong-Wha Chong,et al.  Chirp Spread Spectrum Transceiver Design and Implementation for Real Time Locating System , 2015, Int. J. Distributed Sens. Networks.

[63]  Daniel Gatica-Perez,et al.  StressSense: detecting stress in unconstrained acoustic environments using smartphones , 2012, UbiComp.

[64]  Lei Yang,et al.  AudioGest: enabling fine-grained hand gesture detection by decoding echo signal , 2016, UbiComp.

[65]  Nam Soo Kim,et al.  Acoustic Data Transmission Based on Modulated Complex Lapped Transform , 2010, IEEE Signal Processing Letters.

[66]  Huihuang Zheng,et al.  High-precision acoustic motion tracking: demo , 2016, MobiCom.

[67]  S.E. El-Khamy,et al.  Efficient multiple-access communications using multi-user chirp modulation signals , 1996, Proceedings of ISSSTA'95 International Symposium on Spread Spectrum Techniques and Applications.

[68]  Xinyu Zhang,et al.  Autodirective audio capturing through a synchronized smartphone array , 2014, MobiSys.

[69]  Cristina Videira Lopes,et al.  Aerial acoustic communications , 2001, Proceedings of the 2001 IEEE Workshop on the Applications of Signal Processing to Audio and Acoustics (Cat. No.01TH8575).

[70]  Yunhao Liu,et al.  Lasagna: towards deep hierarchical understanding and searching over mobile sensing data , 2016, MobiCom.

[71]  Rong Zheng,et al.  ARABIS: An asynchronous acoustic indoor positioning system for mobile devices , 2017, 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[72]  Jie Yang,et al.  Snooping Keystrokes with mm-level Audio Ranging on a Single Phone , 2015, MobiCom.

[73]  Swarun Kumar,et al.  Decimeter-Level Localization with a Single WiFi Access Point , 2016, NSDI.

[74]  Scott Counts,et al.  Supporting social presence through lightweight photo sharing on and off the desktop , 2004, CHI.

[75]  Wei Wang,et al.  Device-free gesture tracking using acoustic signals , 2016, MobiCom.

[76]  Alan L. Yuille,et al.  An Approach to Pose-Based Action Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[77]  Richard M. White,et al.  Acoustic sensors for physical, chemical and biochemical applications , 1998, Proceedings of the 1998 IEEE International Frequency Control Symposium (Cat. No.98CH36165).

[78]  Guobin Shen,et al.  BeepBeep: a high accuracy acoustic ranging system using COTS mobile devices , 2007, SenSys '07.

[79]  Yunhao Liu,et al.  Context-free Attacks Using Keyboard Acoustic Emanations , 2014, CCS.

[80]  Wenyuan Xu,et al.  DolphinAttack: Inaudible Voice Commands , 2017, CCS.

[81]  Xiaoli Ma,et al.  Robust synchronization for OFDM employing Zadoff-Chu sequence , 2012, 2012 46th Annual Conference on Information Sciences and Systems (CISS).

[82]  Sangki Yun,et al.  Strata: Fine-Grained Acoustic-based Device-Free Tracking , 2017, MobiSys.

[83]  Eric C. Larson,et al.  Accurate and privacy preserving cough sensing using a low-cost microphone , 2011, UbiComp '11.

[84]  Walter Bender,et al.  Things that talk: Using sound for device-to-device and device-to-human communication , 2000, IBM Syst. J..

[85]  John Terry,et al.  OFDM Wireless LANs: A Theoretical and Practical Guide , 2001 .

[86]  Cecilia Mascolo,et al.  EmotionSense: a mobile phones based adaptive platform for experimental social psychology research , 2010, UbiComp.

[87]  Allan Kuchinsky,et al.  Requirements for photoware , 2002, CSCW '02.

[88]  Muhammad Shahzad,et al.  Position and Orientation Agnostic Gesture Recognition Using WiFi , 2017, MobiSys.

[89]  David Tse,et al.  Fundamentals of Wireless Communication , 2005 .

[90]  Philippe Lacomme,et al.  Air and Spaceborne Radar Systems: An Introduction , 2001 .

[91]  Sachin Katti,et al.  SpotFi: Decimeter Level Localization Using WiFi , 2015, SIGCOMM.

[92]  Sangki Yun,et al.  Turning a Mobile Device into a Mouse in the Air , 2015, MobiSys.

[93]  Ming Yang,et al.  DeepFace: Closing the Gap to Human-Level Performance in Face Verification , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[94]  Sangki Yun,et al.  Indoor Follow Me Drone , 2017, MobiSys.

[95]  Kris M. Kitani,et al.  Going Deeper into First-Person Activity Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[96]  Roozbeh Jafari,et al.  Robust activity recognition using wearable IMU sensors , 2014, IEEE SENSORS 2014 Proceedings.

[97]  Shwetak N. Patel,et al.  Whole-home gesture recognition using wireless signals , 2013, MobiCom.

[98]  Kristof Van Laerhoven,et al.  Real-time Embedded Recognition of Sign Language Alphabet Fingerspelling in an IMU-Based Glove , 2017, iWOAR.

[99]  Khaled A. Harras,et al.  WiGest: A ubiquitous WiFi-based gesture recognition system , 2014, 2015 IEEE Conference on Computer Communications (INFOCOM).

[100]  Mohamed Ibnkahla,et al.  Principles of MIMO-OFDM Wireless Systems , 2004 .

[101]  Sugata Sanyal,et al.  Survey of Security and Privacy Issues of Internet of Things , 2015, ArXiv.

[102]  Xinyu Zhang,et al.  Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization , 2014, MobiSys.

[103]  Kaushik Mahata,et al.  Zadoff-Chu sequence design for random access initial uplink synchronization , 2016, 1604.01476.