SilentSpeller: Towards mobile, hands-free, silent speech text entry using electropalatography

Speech is inappropriate in many situations, limiting when voice control can be used. Most unvoiced speech text entry systems can not be used while on-the-go due to movement artifacts. Using a dental retainer with capacitive touch sensors, SilentSpeller tracks tongue movement, enabling users to type by spelling words without voicing. SilentSpeller achieves an average 97% character accuracy in offline isolated word testing on a 1164-word dictionary. Walking has little effect on accuracy; average offline character accuracy was roughly equivalent on 107 phrases entered while walking (97.5%) or seated (96.5%). To demonstrate extensibility, the system was tested on 100 unseen words, leading to an average 94% accuracy. Live text entry speeds for seven participants averaged 37 words per minute at 87% accuracy. Comparing silent spelling to current practice suggests that SilentSpeller may be a viable alternative for silent mobile text entry.

[1]  T. Starner,et al.  Typing on Tap: Estimating a Finger-Worn One-Handed Chording Keyboard’s Text Entry Rate , 2021, SEMWEB.

[2]  Jun Rekimoto,et al.  Mobile, Hands-free, Silent Speech Texting Using SilentSpeller , 2021, CHI Extended Abstracts.

[3]  Ahmed Sabbir Arif,et al.  LipType: A Silent Speech Recognizer Augmented with an Independent Repair Model , 2021, CHI.

[4]  Meredith Ringel Morris,et al.  Social Media through Voice , 2021, Proc. ACM Hum. Comput. Interact..

[5]  Zhanpeng Jin,et al.  EchoWhisper , 2020, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[6]  Zhanpeng Jin,et al.  EchoWhisper , 2020 .

[7]  Peter Birkholz,et al.  Cross-Speaker Silent-Speech Command Word Recognition Using Electro-Optical Stomatography , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[8]  Pattie Maes,et al.  Non-Invasive Silent Speech Recognition in Multiple Sclerosis with Dysphonia , 2020, ML4H@NeurIPS.

[9]  Swarun Kumar,et al.  RFID Tattoo , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[10]  Jacob O. Wobbrock,et al.  Beyond the Input Stream: Making Text Entry Evaluations More Flexible with Transcription Sequences , 2019, UIST.

[11]  Per Ola Kristensson,et al.  How do People Type on Mobile Devices?: Observations from a Study with 37,000 Volunteers , 2019, MobileHCI.

[12]  Jun Rekimoto,et al.  SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks , 2019, CHI.

[13]  Quoc V. Le,et al.  SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition , 2019, INTERSPEECH.

[14]  Jason Wu,et al.  TongueBoard: An Oral Interface for Subtle Input , 2019, AH.

[15]  Jacob O. Wobbrock,et al.  Situationally-Induced Impairments and Disabilities , 2019, Web Accessibility.

[16]  Yuanchun Shi,et al.  Lip-Interact: Improving Mobile Device Interaction with Silent Speech Commands , 2018, UIST.

[17]  Masaaki Fukumoto,et al.  SilentVoice: Unnoticeable Voice Input by Ingressive Speech , 2018, UIST.

[18]  Gábor Gosztolya,et al.  Multi-Task Learning of Speech Recognition and Speech Synthesis Parameters for Ultrasound-based Silent Speech Interfaces , 2018, INTERSPEECH.

[19]  Yunbin Deng,et al.  Development of sEMG sensors and algorithms for silent speech recognition , 2018, Journal of neural engineering.

[20]  Dong Sup Lee,et al.  Wireless, intraoral hybrid electronics for real-time quantification of sodium intake toward hypertension management , 2018, Proceedings of the National Academy of Sciences.

[21]  Gábor Gosztolya,et al.  F0 Estimation for DNN-Based Ultrasound Silent Speech Interfaces , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[22]  Pattie Maes,et al.  AlterEgo: A Personalized Wearable Silent Speech Interface , 2018, IUI.

[23]  Bruce Denby,et al.  Updating the silent speech challenge benchmark with deep learning , 2017, Speech Commun..

[24]  James A. Landay,et al.  Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones , 2016, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[25]  Gábor Gosztolya,et al.  DNN-Based Ultrasound-to-Speech Conversion for a Silent Speech Interface , 2017, INTERSPEECH.

[26]  Michael Gaihede,et al.  Development and functional demonstration of a wireless intraoral inductive tongue computer interface for severely disabled persons , 2017, Disability and rehabilitation. Assistive technology.

[27]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[28]  Laurent Girin,et al.  Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces , 2016, PLoS Comput. Biol..

[29]  Phil D. Green,et al.  A silent speech system based on permanent magnet articulography and direct synthesis , 2016, Comput. Speech Lang..

[30]  Samuel S. Silva,et al.  An Introduction to Silent Speech Interfaces , 2016 .

[31]  Maysam Ghovanloo,et al.  Toward Silent-Speech Control of Consumer Wearables , 2015, Computer.

[32]  Tzyy-Ping Jung,et al.  High-speed spelling with a noninvasive brain–computer interface , 2015, Proceedings of the National Academy of Sciences.

[33]  Li Liu,et al.  Tongue-able interfaces: evaluating techniques for a camera based tongue gesture input system , 2014, ASSETS.

[34]  David P. Quigley,et al.  Texting while walking: an evaluation of mini-qwerty text input while on-the-go , 2014, MobileHCI '14.

[35]  I. Scott MacKenzie,et al.  Applying Small-Keyboard Computer Control to the Real World , 2014, ICCHP.

[36]  Héctor A. Caltenco,et al.  On the tip of the tongue: learning typing and pointing with an intra-oral computer interface , 2014, Disability and rehabilitation. Assistive technology.

[37]  Andrew W. Senior,et al.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling , 2014, INTERSPEECH.

[38]  Pedro Lopes,et al.  Gesture output: eyes-free output using a force feedback touch surface , 2013, CHI.

[39]  Shwetak N. Patel,et al.  GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones , 2012, UIST.

[40]  Shumin Zhai,et al.  The word-gesture keyboard: reimagining keyboard interaction , 2012, CACM.

[41]  Jacob O. Wobbrock,et al.  WalkType: using accelerometer data to accomodate situational impairments in mobile touch screen text entry , 2012, CHI.

[42]  Krzysztof Z. Gajos,et al.  Ability-Based Design: Concept, Principles and Examples , 2011, TACC.

[43]  Sercan Taha Ahi,et al.  A Dictionary-Driven P300 Speller With a Modified Interface , 2011, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[44]  Tanja Schultz,et al.  ICCHP Keynote: Recognizing Silent and Weak Speech Based on Electromyography , 2010, ICCHP.

[45]  Tomoki Toda,et al.  Silent-speech enhancement using body-conducted vocal-tract resonance signals , 2010, Speech Commun..

[46]  J. M. Gilbert,et al.  Silent speech interfaces , 2010, Speech Commun..

[47]  Gérard Chollet,et al.  Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips , 2010, Speech Commun..

[48]  Carlos Hitoshi Morimoto,et al.  Context switching for fast key selection in text entry applications , 2010, ETRA.

[49]  John Paulin Hansen,et al.  Evaluation of a low-cost open-source gaze tracker , 2010, ETRA.

[50]  Oleg Spakov,et al.  Fast gaze typing with an adjustable dwell time , 2009, CHI.

[51]  J. M. Gilbert,et al.  Development of a (silent) speech recognition system for patients following laryngectomy. , 2008, Medical engineering & physics.

[52]  I. Scott MacKenzie,et al.  Graffiti vs. unistrokes: an empirical comparison , 2008, CHI.

[53]  Poika Isokoski,et al.  Now Dasher! Dash away!: longitudinal study of fast text entry by Eye Gaze , 2008, ETRA.

[54]  Jacob O. Wobbrock,et al.  Longitudinal evaluation of discrete consecutive gaze gestures for text entry , 2008, ETRA.

[55]  Gérard Chollet,et al.  Eigentongue Feature Extraction for an Ultrasound-Based Silent Speech Interface , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[56]  Kumiko Tanaka-Ishii,et al.  Text Entry Systems: Mobility, Accessibility, Universality , 2007 .

[57]  Yohan Payan,et al.  Guiding the Surgical Gesture Using an Electro-Tactile Stimulus Array on the Tongue: A Feasibility Study , 2007, IEEE Transactions on Biomedical Engineering.

[58]  Jacob O. Wobbrock,et al.  Measures of Text Entry Performance , 2007 .

[59]  Brad A. Myers,et al.  Edgewrite: a versatile design for text entry and control , 2006 .

[60]  Kent Lyons,et al.  The impacts of limited visual feedback on mobile text entry for the Twiddler and mini-QWERTY keyboards , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[61]  Kent Lyons,et al.  An empirical study of typing rates on mini-QWERTY keyboards , 2005, CHI Extended Abstracts.

[62]  Kent Lyons,et al.  Twiddler typing: one-handed chording text entry for mobile phones , 2004, CHI.

[63]  Lawrence R. Rabiner,et al.  Automatic Speech Recognition - A Brief History of the Technology Development , 2004 .

[64]  Tracy L. Westeyn,et al.  Georgia tech gesture toolkit: supporting experiments in gesture recognition , 2003, ICMI '03.

[65]  Brad A. Myers,et al.  EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion , 2003, UIST '03.

[66]  Kiyohiro Shikano,et al.  Non-audible murmur recognition input interface using stethoscopic microphone attached to the skin , 2003, 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03)..

[67]  I. Scott MacKenzie,et al.  Metrics for text entry research: an evaluation of MSD and KSPC, and a new unified error metric , 2003, CHI '03.

[68]  I. Scott MacKenzie,et al.  Phrase sets for evaluating text entry techniques , 2003, CHI Extended Abstracts.

[69]  J. Picone,et al.  Continuous speech recognition using hidden Markov models , 1990, IEEE ASSP Magazine.

[70]  W. Hardcastle,et al.  New developments in electropalatography: A state-of-the-art report , 1989 .