RFID Tattoo

This paper presents an RF-based assistive technology for voice impairments (i.e., dysphonia), which occurs in an estimated 1% of the global population. We specifically focus on acquired voice disorders where users continue to be able to make facial and lip gestures associated with speech. Despite the rich literature on assistive technologies in this space, there remains a gap for a solution that neither requires external infrastructure in the environment, battery-powered sensors on skin or body-worn manual input devices. We present RFTattoo, which to our knowledge is the first wireless speech recognition system for voice impairments using batteryless and flexible RFID tattoos. We design specialized wafer-thin tattoos attached around the user's face and easily hidden by makeup. We build models that process signal variations from these tattoos to a portable RFID reader to recognize various facial gestures corresponding to distinct classes of sounds. We then develop natural language processing models that infer meaningful words and sentences based on the observed series of gestures. A detailed user study with 10 users reveals 86% accuracy in reconstructing the top-100 words in the English language, even without the users making any sounds.

[1]  Joshua R. Smith,et al.  Neuralwisp: a Wirelessly Powered Neural Interface with 1-m Range , 2022 .

[2]  Eric Paulos,et al.  Skintillates: Designing and Creating Epidermal Interactions , 2016, Conference on Designing Interactive Systems.

[3]  Swarun Kumar,et al.  RF-Wear: Towards Wearable Everyday Skeleton Tracking Using Passive RFIDs , 2018, UbiComp/ISWC Adjunct.

[4]  Gerasimos Potamianos,et al.  Exploring ROI size in deep learning based lipreading , 2017, AVSP.

[5]  Luca Catarinucci,et al.  Smart Prototyping Techniques for UHF RFID Tags: Electromagnetic Characterization and Comparison with Traditional Approaches , 2012 .

[6]  Naomi Harte,et al.  Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition , 2018, ICMI.

[7]  Kaishun Wu,et al.  We Can Hear You with Wi-Fi! , 2016, IEEE Trans. Mob. Comput..

[8]  Charles Jorgensen,et al.  Small Vocabulary Recognition Using Surface Electromyography in an Acoustically Harsh Environment , 2005 .

[9]  Masaaki Fukumoto,et al.  SilentVoice: Unnoticeable Voice Input by Ingressive Speech , 2018, UIST.

[10]  Unsoo Ha,et al.  Learning Food Quality and Safety from Wireless Stickers , 2018, HotNets.

[11]  W. Kim,et al.  Stretchable RFID for Wireless Strain Sensing With Silver Nano Ink , 2014, IEEE Sensors Journal.

[12]  Maja Pantic,et al.  Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[13]  Fadel Adib,et al.  Minding the Billions: Ultra-wideband Localization for Deployed RFID Tags , 2017, MobiCom.

[14]  Ju Wang,et al.  TagScan: Simultaneous Target Imaging and Material Identification with Commodity RFID Devices , 2017, MobiCom.

[15]  Devin J. Balkcom,et al.  Reconstructing Human Joint Motion with Computational Fabrics , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[16]  Alex Olwal,et al.  SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics , 2017, CHI.

[17]  MARTHA MBENGE.  INTEREFERENCE CANCELLATION IN RFID SYSTEMS , 2016 .

[18]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[19]  Dongsuk Yook,et al.  Audio-to-Visual Conversion Using Hidden Markov Models , 2002, PRICAI.

[20]  A. Azzouz.  2011 , 2020, City.

[21]  I. Elamvazuthi,et al.  Voice Recognition Algorithms using Mel Frequency Cepstral Coefficient (MFCC) and Dynamic Time Warping (DTW) Techniques , 2010, ArXiv.

[22]  Richard Harvey,et al.  Decoding visemes: Improving machine lip-reading , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[23]  Yuhao Liu,et al.  Lab-on-Skin: A Review of Flexible and Stretchable Electronics for Wearable Health Monitoring. , 2017, ACS nano.

[24]  Kaishun Wu,et al.  We Can Hear You with Wi-Fi! , 2014, IEEE Transactions on Mobile Computing.

[25]  Cristian Danescu-Niculescu-Mizil,et al.  Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs , 2011, CMCL@ACL.

[26]  Joon Son Chung,et al.  Lip Reading Sentences in the Wild , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  P Suppes,et al.  Brain wave recognition of words. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Karthikeyan Sundaresan,et al.  RIO: A Pervasive RFID-based Touch Gesture Interface , 2017, MobiCom.

[29]  Carmel Majidi,et al.  Chasing biomimetic locomotion speeds: Creating untethered soft robots with shape memory alloy actuators , 2018, Science Robotics.

[30]  Ranvijay,et al.  Lip reading techniques: A survey , 2016, 2016 2nd International Conference on Applied and Theoretical Computing and Communication Technology (iCATccT).

[31]  Matthias Janke,et al.  EMG-to-Speech: Direct Generation of Speech From Facial Electromyographic Signals , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[32]  Christian Holz,et al.  DuoSkin: rapidly prototyping on-skin user interfaces using skin-friendly materials , 2016, SEMWEB.

[33]  John C. Batchelor,et al.  Effect of skin dielectric properties on the read range of epidermal ultra-high frequency radio-frequency identification tags , 2017, Healthcare technology letters.

[34]  M.M. Nice,et al.  Length of sentences as a criterion of a child’s progress in speech , 1925 .

[35]  C. G. Fisher,et al.  Confusions among visually perceived consonants. , 1968, Journal of speech and hearing research.

[36]  A. Mihailidis,et al.  Difficulties in Automatic Speech Recognition of Dysarthric Speakers and Implications for Speech-Based Applications Used by the Elderly: A Literature Review , 2010, Assistive technology : the official journal of RESNA.

[37]  Lei Yang,et al.  Tagoram: real-time tracking of mobile RFID tags to high precision using COTS devices , 2014, MobiCom.

[38]  Helen L. Bear.  Decoding visemes: improving machine lipreading (PhD thesis) , 2017, ArXiv.

[39]  P. Bhartia,et al.  Millimeter-wave microstrip and printed circuit antennas , 1991 .

[40]  Swarun Kumar,et al.  Pushing the Range Limits of Commercial Passive RFIDs , 2019, NSDI.

[41]  G. Marrocco,et al.  Passive RFID Strain-Sensor Based on Meander-Line Antennas , 2011, IEEE Transactions on Antennas and Propagation.

[42]  Maysam Ghovanloo,et al.  Toward Silent-Speech Control of Consumer Wearables , 2015, Computer.

[43]  Jung Woo Lee,et al.  Rugged and breathable forms of stretchable electronics with adherent composite substrates for transcutaneous monitoring , 2014, Nature Communications.

[44]  Jon Barker,et al.  An audio-visual corpus for speech perception and automatic speech recognition. , 2006, The Journal of the Acoustical Society of America.

[45]  Tong Lu,et al.  iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing , 2015, CHI.

[46]  Jinlan Gao,et al.  Printed Electromagnetic Coupler With an Embedded Moisture Sensor for Ordinary Passive RFID Tags , 2011, IEEE Electron Device Letters.

[47]  Markus P. Nemitz,et al.  Soft Radio-Frequency Identification Sensors: Wireless Long-Range Strain Sensors Using Radio-Frequency Identification , 2019, Soft robotics.

[48]  Wen-Jing Yan,et al.  The Dynamic Features of Lip Corners in Genuine and Posed Smiles , 2018, Front. Psychol..

[49]  Manos M. Tentzeris,et al.  A novel chipless RFID-based stretchable and wearable hand gesture sensor , 2015, 2015 European Microwave Conference (EuMC).

[50]  Carmel Majidi,et al.  Liquid Metal-Microelectronics Integration for a Sensorized Soft Robot Skin , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[51]  C. Pyo,et al.  Design of RFID tag antennas using an inductively coupled feed , 2005 .

[52]  B. V. K. Vijaya Kumar,et al.  Imagined Speech Classification with EEG Signals for Silent Communication: A Preliminary Investigation into Synthetic Telepathy , 2010, 2010 4th International Conference on Bioinformatics and Biomedical Engineering.

[53]  Filippo Costa,et al.  Chipless RFID trasponders by using multi-resonant High-Impedance Surfaces , 2013, 2013 International Symposium on Electromagnetic Theory.

[54]  Carmel Majidi,et al.  An autonomously electrically self-healing liquid metal–elastomer composite for robust soft-matter robotics and electronics , 2018, Nature Materials.

[55]  Omid Salehi-Abari,et al.  Challenge: RFID Hacking for Fun and Profit , 2018, MobiCom.

[56]  D. Jackson,et al.  Analysis of planar strip geometries in a substrate-superstrate configuration , 1986 .

[57]  Jürgen Steimle,et al.  Tacttoo: A Thin and Feel-Through Tattoo for On-Skin Tactile Output , 2018, UIST.

[58]  Haiying Huang,et al.  Exploiting a patch antenna for strain measurements , 2008 .

[59]  Filip Ilievski,et al.  Multigait soft robot , 2011, Proceedings of the National Academy of Sciences.

[60]  Swarun Kumar,et al.  WiSh: Towards a Wireless Shape-aware World using Passive RFIDs , 2018, MobiSys.

[61]  Bo Gao,et al.  A System-on-Chip EPC Gen-2 Passive UHF RFID Tag With Embedded Temperature Sensor , 2010, IEEE Journal of Solid-State Circuits.

[62]  Shimon Whiteson,et al.  LipNet: Sentence-level Lipreading , 2016, ArXiv.

[63]  Toshiaki Sugimura,et al.  "Unvoiced speech recognition using EMG - mime speech recognition" , 2003, CHI Extended Abstracts.

[64]  Gierad Laput,et al.  ElectroDermis: Fully Untethered, Stretchable, and Highly-Customizable Electronic Bandages , 2019, CHI.

[65]  Pattie Maes,et al.  AlterEgo: A Personalized Wearable Silent Speech Interface , 2018, IUI.

[66]  Ahmad Basheer Hassanat,et al.  Visual Words for Automatic Lip-Reading , 2014, ArXiv.