PrivacyMic: Utilizing Inaudible Frequencies for Privacy Preserving Daily Activity Recognition

Sound presents an invaluable signal source that enables computing systems to perform daily activity recognition. However, microphones are optimized for human speech and hearing ranges: capturing private content, such as speech, while omitting useful, inaudible information that can aid in acoustic recognition tasks. We simulated acoustic recognition tasks using sounds from 127 everyday household/workplace objects, finding that inaudible frequencies can act as a substitute for privacy-sensitive frequencies. To take advantage of these inaudible frequencies, we designed a Raspberry Pi-based device that captures inaudible acoustic frequencies with settings that can remove speech or all audible frequencies entirely. We conducted a perception study, where participants “eavesdropped’’ on PrivacyMic’s filtered audio and found that none of our participants could transcribe speech. Finally, PrivacyMic’s real-world activity recognition performance is comparable to our simulated results, with over 95% classification accuracy across all environments, suggesting immediate viability in performing privacy-preserving daily activity recognition.

[1]  Norihisa Segawa,et al.  Multiple door opening/closing detection system using infrasound sensor: poster abstract , 2018, IPSN.

[2]  Ray-I Chang,et al.  Query-Based Machine Learning Model for Data Analysis of Infrasonic Signals in Wireless Sensor Networks , 2018, ICDSP 2018.

[3]  Richard F. Lyon,et al.  Ultrasonic Communication Using Consumer Hardware , 2018, IEEE Transactions on Multimedia.

[4]  Subbarao Kambhampati,et al.  What's up with Privacy?: User Preferences and Privacy Concerns in Intelligent Personal Assistants , 2017, AIES.

[5]  Eric C. Larson,et al.  HydroSense: infrastructure-mediated single-point sensing of whole-home water activity , 2009, UbiComp.

[6]  Shwetak N. Patel,et al.  ElectriSense: single-point sensing using EMI for electrical event detection and classification in the home , 2010, UbiComp.

[7]  Bruno Bouchard,et al.  Real-Time Indoor Localization in Smart Homes Using Ultrasound Technology , 2016, PETRA.

[8]  Xavier Serra,et al.  Freesound technical demo , 2013, ACM Multimedia.

[9]  I. Martínez,et al.  Early hominin auditory capacities , 2015, Science Advances.

[10]  Gierad Laput,et al.  Synthetic Sensors: Towards General-Purpose Sensing , 2017, CHI.

[11]  Konrad Rieck,et al.  Privacy Threats through Ultrasonic Side Channels on Mobile Devices , 2017, 2017 IEEE European Symposium on Security and Privacy (EuroS&P).

[12]  Bjoern H. Menze,et al.  A comparison of random forest and its Gini importance with standard chemometric methods for the feature selection and classification of spectral data , 2009, BMC Bioinformatics.

[13]  Ingo R. Titze,et al.  Principles of voice production , 1994 .

[14]  Jilong Kuang,et al.  Configurable Pulmonary-Tuned Privacy Preservation Algorithm for Mobile Devices , 2018, 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[15]  Chris Harrison,et al.  Interferi: Gesture Sensing using On-Body Acoustic Interferometry , 2019, CHI.

[16]  S. M. Williams,et al.  The Audible Spectrum , 2001 .

[17]  Heikki Huttunen,et al.  Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[18]  Wilmot Li,et al.  Lamello: Passive Acoustic Sensing for Tangible Input Components , 2015, CHI.

[19]  Aren Jansen,et al.  Audio Set: An ontology and human-labeled dataset for audio events , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[20]  Vesa T. Peltonen,et al.  Audio-based context recognition , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[21]  Kevin Fu,et al.  Light Commands: Laser-Based Audio Injection Attacks on Voice-Controllable Systems , 2020, USENIX Security Symposium.

[22]  Gierad Laput,et al.  Ubicoustics: Plug-and-Play Acoustic Activity Recognition , 2018, UIST.

[23]  Romit Roy Choudhury,et al.  BackDoor: Making Microphones Hear Inaudible Sounds , 2017, MobiSys.

[24]  Nicolai Petkov,et al.  Reliable detection of audio events in highly noisy environments , 2015, Pattern Recognit. Lett..

[25]  Chris Harrison,et al.  BeamBand: Hand Gesture Sensing with Ultrasonic Beamforming , 2019, CHI.

[26]  Gregory D. Abowd,et al.  FingerPing: Recognizing Fine-grained Hand Poses using Active Acoustic On-body Sensing , 2018, CHI.

[27]  Mike Fraser,et al.  EchoFlex: Hand Gesture Recognition using Ultrasound Imaging , 2017, CHI.

[28]  Eric C. Larson,et al.  Accurate and privacy preserving cough sensing using a low-cost microphone , 2011, UbiComp '11.

[29]  Wenyao Xu,et al.  PDVocal: Towards Privacy-preserving Parkinson's Disease Detection using Non-speech Body Sounds , 2019, MobiCom.

[30]  Yvan Saeys,et al.  Robust Feature Selection Using Ensemble Feature Selection Techniques , 2008, ECML/PKDD.

[31]  Bhiksha Raj,et al.  NELS - Never-Ending Learner of Sounds , 2018, ArXiv.

[32]  Eric C. Larson,et al.  SpiroCall: Measuring Lung Function over a Phone Call , 2016, CHI.

[33]  Antonio Torralba,et al.  SoundNet: Learning Sound Representations from Unlabeled Video , 2016, NIPS.